Analyzing 1.1 Billion NYC Taxi and Uber Trips, with a Vengeance

Over at Todd W. Schneider's website:

NYC Taxi Data

The official TLC trip record dataset contains data for over 1.1 billion taxi trips from January 2009 through June 2015, covering both yellow and green taxis. Each individual trip record contains precise location coordinates for where the trip started and ended, timestamps for when the trip started and ended, plus a few other variables including fare amount, payment method, and distance traveled.

I used PostgreSQL to store the data and PostGIS to perform geographic calculations, including the heavy lifting of mapping latitude/longitude coordinates to NYC census tracts and neighborhoods. The full dataset takes up 267 GB on disk, before adding any indexes. For more detailed information on the database schema and geographic calculations, take a look at the GitHub repository.

Uber Data

Thanks to the folks at FiveThirtyEight, there is also some publicly available data covering nearly 19 million Uber rides in NYC from April–September 2014 and January–June 2015, which I’ve incorporated into the dataset. The Uber data is not as detailed as the taxi data, in particular Uber provides time and location for pickups only, not drop offs, but I wanted to provide a unified dataset including all available taxi and Uber data. Each trip in the dataset has a cab_type_id, which indicates whether the trip was in a yellow taxi, green taxi, or Uber car.

The introduction of the green boro taxi program in August 2013 dramatically increased the amount of taxi activity in the outer boroughs. Here’s a graph of taxi pickups in Brooklyn, the most populous borough, split by cab type:

brooklyn pickups

From 2009–2013, a period during which migration from Manhattan to Brooklyn generally increased, yellow taxis nearly doubled the number of pickups they made in Brooklyn.

Once boro taxis appeared on the scene, though, the green taxis quickly overtook yellow taxis so that as of June 2015, green taxis accounted for 70% of Brooklyn’s 850,000 monthly taxi pickups, while yellow taxis have decreased Brooklyn pickups back to their 2009 rate. Yellow taxis still account for more drop offs in Brooklyn, since many people continue to take taxis from Manhattan to Brooklyn, but even in drop offs, the green taxis are closing the gap.

More here.

The Algorithm That Creates Diets That Work for You

Ed Yong in The Atlantic:

DaalTake a slice of cake and cut it in two. Eat one half, and let a friend scoff the other. Your blood-sugar levels will both spike, but to different degrees depending on your genes, the bacteria in your gut, what you recently ate, how recently or intensely you exercised, and more. The spikes, formally known as “postprandial glycemic responses” or PPGR, are hard to forecast since two people might react very differently to exactly the same food.

But Eran Elinav and Eran Segal from the Weizmann Institute of Science have developed a way of embracing that variability. By comprehensively monitoring the blood sugar, diets, and other traits of 800 people, they built an algorithm that can accurately predict how a person's blood-sugar levels will spike after eating any given meal.

They also used these personalized predictions to develop tailored dietary plans for keeping blood sugar in check. These plans sometimes included unconventional items like chocolate and ice-cream, and were so counter-intuitive that they baffled both the participants and dieticians involved in the study. But they seemed to work when assessed in a clinical trial, and they hint at a future when individuals will get personalized dietary recommendations, rather than hewing to universal guidelines.

More here.

English really is weirder than pretty much every other language

John McWhorter in Aeon:

ScreenHunter_1513 Nov. 25 17.58English speakers know that their language is odd. So do people saddled with learning it non-natively. The oddity that we all perceive most readily is its spelling, which is indeed a nightmare. In countries where English isn’t spoken, there is no such thing as a ‘spelling bee’ competition. For a normal language, spelling at least pretends a basic correspondence to the way people pronounce the words. But English is not normal.

Spelling is a matter of writing, of course, whereas language is fundamentally about speaking. Speaking came long before writing, we speak much more, and all but a couple of hundred of the world’s thousands of languages are rarely or never written. Yet even in its spoken form, English is weird. It’s weird in ways that are easy to miss, especially since Anglophones in the United States and Britain are not exactly rabid to learn other languages. But our monolingual tendency leaves us like the proverbial fish not knowing that it is wet. Our language feels ‘normal’ only until you get a sense of what normal really is.

There is no other language, for example, that is close enough to English that we can get about half of what people are saying without training and the rest with only modest effort. German and Dutch are like that, as are Spanish and Portuguese, or Thai and Lao.

More here.

The value of math in a modern world

Greg Rienzi in HUB:

ScreenHunter_1512 Nov. 25 17.51Math as both profession and course of study can be a hard sell, something even Don Draper might have trouble pitching. The field unites numbers, theories, and ideas that, yes, can be physically represented but remain intangible. Math is a language unto itself that for some might as well be Latin or Klingon. Even its rare turns in popular culture—A Beautiful Mind,Proof, and The Big Bang Theory come to mind—typically depict brilliant but troubled and/or socially handicapped thinkers more absorbed by theory than reality.

To Richard Brown, however, math can be as beautiful as a ray of morning sunlight cast upon an orchid's petals, as lyrical as a Beethoven symphony. “Math is not about the numbers,” says Brown, director of undergraduate studies in Department of Mathematics at Johns Hopkins University. “It's the ideas behind the numbers. Yes, you can say it's built on a rigid set of rules, but out of that comes an infinite amount of creativity.” And, he adds, beauty.

In recent years, Brown has served as math spokesman of sorts. Last year, he presented a talk at the inaugural TedxJohnsHopkinsUniversity titled “Why Mathematics?”. He opened up with a question: “Why on earth would anyone choose math as a field of discipline to study, or construct a career in the field?”

More here.

A sensory illusion that makes yeast cells self-destruct: possible tactic for cancer therapeutics

From KurzweiAI:

Osmotic-oscillationsUC San Francisco researchers have discovered that even brainless single-celled yeast have “sensory biases” that can be hacked by a carefully engineered illusion — a finding that could be used to develop new approaches to fighting diseases such as cancer. In the new study, published online Thursday November 19 in Science Express, Wendell Lim, PhD, the study’s senior author*, and his team discovered that yeast cells falsely perceive a pattern of osmotic levels (by applying potassium chloride) that alternate in eight minute intervals as massive, continuously increasing stress. In response, the microbes over-respond and kill themselves. (In their natural environment, salt stress normally gradually increases.) The results, Lim says, suggest a whole new way of looking at the perceptual abilities of simple cells and this power of illusion could even be used to develop new approaches to fighting cancer and other diseases.

“Our results may also be relevant for cellular signaling in disease, as mutations affecting cellular signaling are common in cancer, autoimmune disease, and diabetes,” the researchers conclude in the paper. “These mutations may rewire the native network, and thus could modify its activation and adaptation dynamics. Such network rewiring in disease may lead to changes that can be most clearly revealed by simulation with oscillatory inputs or other ‘non-natural’ patterns. “The changes in network response behaviors could be exploited for diagnosis and functional profiling of disease cells, or potentially taken advantage of as an Achilles’ heel to selectively target cells bearing the diseased network.”

More here.

A Grammar of Pain

Jacqueline Feldman in Harvard Magazine:

OMAW_image_1_HD_copyright_AliceFilmsAs Of Men and War opens, a trumpet moans and won’t resolve into taps, and a van traps men who let it bounce them. They rarely come to town, they say. They are quick to road rage. They wear sunglasses. Later, one man tells his wife, “If we never had to leave the house, I’d be all right, but it’s going outside I can’t handle too much.” The French documentary (released in many U.S. cities this week after playing at theaters in France and at international festivals), follows veterans of Iraq and Afghanistan as they undergo trauma therapy at a residential center called Pathway Home, founded by therapist Fred Gusman. These characters are archetypal—“almost mythological”—says Isidore Bethel ’11, the head editor for the film and one of its associate producers: “These men could have served in any war.” Their transposition into an older pantheon of suffering is nimble. The contemporary term “PTSD” is used exactly twice, once by a vet guilt-tripping a too-persistent caller and then by another vet’s wife. Otherwise, no character mentions diagnoses or pills, only men and death and their closeness to it.

Of Men and War is the second film in Laurent Bécue-Renard’s series A Genealogy of Wrath. As a young man, the director spent several months in besieged Sarajevo for “purely romantic” reasons. He found the city’s high pitch catching and vaguely familiar. There, while writing news dispatches and short stories, he met a therapist who worked with war widows. He focused on three of these women for his first documentary, War-Wearied (2003). It dawned on him that a personal “quest” had drawn him to his project. His grandfathers served in the First World War, which killed some 1.7 million French, about 5 percent of the country’s population. “All of us who grew up in the second half of the twentieth century inherited the legacy of the wars in the first part of the twentieth century,” Bécue-Renard said. “It has been shaping the psyches of the families we belong to.”

More here.

Wednesday Poem

……… Alone together a moment
on the twenty-second anniversary
……… of their wedding
he clasped her as she stood
……… at the sink, pressing
into her backside, rubbing his cheek
……… against the stubble
of her skull. He gave her a ring
……… of pink tourmaline
with nine small diamonds around it.
……… She put it on her finger
and immediately named it Please Don’t Die.
……… They kissed and Jane
whispered, “Timor mortis conturbat me.”

by Donald Hall
from Without
Mariner Books, 1999

Sean Carroll: “What else is possible if space and time can change?”

On the eve of the 100th anniversary of the publication of General Relativity by Albert Einstein, Sean Carroll asks, “Einstein's legacy: if spacetime is dynamical rather than absolute, what else about the universe might be flexible?” at PBS Newshour:

ScreenHunter_1511 Nov. 24 18.46Nicolaus Copernicus is famous for having suggested that the Earth moves around the sun, rather than the other way around. That’s a big deal, as it displaces the Earth from its presumed position at the center of the universe. But it’s easy for us to forget something equally amazing: the idea that the Earth can actually move at all. If anything seems like a solid foundation, it’s the Earth itself. But in our post-Copernican world, we know better.

Albert Einstein, with his general theory of relativity, took this conceptual revolution one step forward. Not only is the Earth not a fixed fulcrum around which the rest of the universe revolves, space and time themselves are not fixed and unchanging. In Einstein’s universe, space and time are absorbed into a single, four-dimensional “spacetime,” and spacetime is not solid. It twists and turns and bends in response to the motion of matter and energy. We perceive that stretching and distortion of the fabric of spacetime as the force of gravity.

The idea that space and time themselves are not immutable, but are dynamical quantities that can evolve through the history of the universe, is one of Einstein’s most dramatic legacies. It was so profound that Einstein himself had trouble accepting all the implications of the idea. When he investigated the universe as a whole in general relativity, he found that it should be expanding or contracting, not staying at a fixed size. That went contrary to his intuition, as well as to what astronomers of the time actually thought the universe was doing. When Edwin Hubble discovered the expansion of the universe in the 1920’s, Einstein realized that he had missed the opportunity to make one of the great predictions in the history of science.

More here.

Goya: The Portraits

Clar05_3722_02T.J. Clark at the London Review of Books:

The longer I looked at the painting the more I was drawn to the dialogue taking place, sotto voce, between the over-coloured count and his shadowy double hung high on the wall above him – a faintly preposterous rehash of the mirror in Las Meninas, where king and queen make their necessary appearance. The dialogue in Goya – the shadow play, the hovering between repetition and caricature – seemed to me to drain both parties (I presumed that the figure on the wall was an ancestor, or maybe the monarch himself) of reality. The second man staring at me – again, a version of a great moment in Las Meninas, where Velázquez in the background fixes his royal sitters with a predatory gaze – seemed to peer from the picture with an expression compounded of alarm, disbelief and sheer uncomfortable consciousness of his place in a game of looking. Looking and being looked at and thereby ‘brought to life’. He’d be damned if he’d occupy the place he’d been allotted. I found myself staring back at the painting in much the same frame of mind. The more I responded to Floridablanca’s local (stunning) reality effects – the silver shimmer on the count’s sash, the light through the glass on the clock face, the spectacles clutched in his fingers, the Zurbarán notebook glowing on the floor – the more it seemed to me they didn’t matter. What mattered – what made the painting Goya’s – was the pervasive unreality of the set-up, swallowing the world of objects and persons no sooner than it conjured them up.

I realise that I haven’t put my finger on what produced the feeling of unreality, and I’m not sure I can. I know there are dangers in trusting the feeling at all. Anyone looking at Goya’s portraits can’t avoid seeing them against the background of the Caprichos and Black Paintings and the unbearable private albums, some drawn, some etched and aquatinted, dwelling on torment and degeneracy.

more here.

“Reality-concealing rhetoric” and our responses to terrorism

23076411945_1bbe7115e0_bPankaj Mishra at n+1:

Fourteen years after September 11, the reality-concealing rhetoric of Westernism participates in a race to extremes with its ideological twin, in an escalated dialectic of bombing from the air and slaughter on the ground. It grows more aggressive in proportion to the spread of the non-West’s chaos to the West, and also blends faster into a white supremacist hatred of immigrants, refugees, and Muslims (and, often, those who just “look” Muslim). Even more menacingly, it postpones the moment of self-reckoning and course-correction among Euro-American elites who seem to have led us, a century after the First World War, into another uncontrollable and extensive conflagration.

Among the more polished examples of their intellectual rearguardism last week was a piece in the Financial Times by the paper’s foreign-affairs columnist, Philip Stephens, titled “Paris attacks must shake Europe’s complacency. The idea that the west should shoulder blame rests on a corrosive moral relativism.”

It should be said that the Financial Times, the preferred newspaper of the Anglo-American intelligentsia as well as Davos Man and his epigones, keeps a fastidious distance, editorially, from the foam-at-the-mouth bellicosity of its direct competitor, the Wall Street Journal (whose op-ed pages often seem to be elaborating on its owner’s demented tweets).

more here.

a forgotten designer

Tea-horns-crop_jpg_600x603_q85Martin Filler at the NYRB:

Among the dozen useful masterpieces chosen by the US Postal Service for its 2011 series of stamps honoring American industrial design pioneers is the Normandie water pitcher (1935), a sleek chromium-plated vessel whose prow-like form echoes that of the then-new French ocean liner for which it was named. This stunning work, manufactured by the Revere Copper and Brass company, evokes the glamour of interwar transatlantic travel with a sculptural purity worthy of Brancusi, and is rightly included in numerous design history books and museum collections. Yet until now the general public has known next to nothing about its creator, Peter Muller-Munk.

That lapse is finally redressed by “Silver to Steel: The Modern Designs of Peter Muller-Munk,” an illuminating exhibition at Pittsburgh’s Carnegie Museum of Art. It vindicates the personal crusade of the Miami-based design historian Jewel Stern, an intrepid researcher of this overlooked figure and the co-curator of the show.

Stern is particularly interested in the transmission of modern design ideas from Europe to the United States, which in previous books she has revealed to be a far more complex and dispersed process than had been thought. Her diligence has paid off handsomely in this fascinating survey, and confirms the unjustly neglected Muller-Munk’s rightful place in the top tier with his much better known American contemporaries Donald Deskey, Henry Dreyfuss, Norman Bel Geddes, Raymond Loewy, and Walter Dorwin Teague.

more here.

This Artist Makes Traditional Carpets That Look Like They Were hit by a Software Bug

Danny Lewis in Smithsonian Magazine:

ScreenHunter_1510 Nov. 24 13.52At first glance, Faig Ahmed’s carpets look like digital photos that didn’t load right the first time you clicked on them. Intricate patterns morph into messes of pixelation; blocks of color slide off like someone scrolled past them too fast; and some of the 2D mats look like they are bulging off a screen. But while they may appear to be software glitches or bad Photoshop editing, every one of Ahmed’s carpets are hand-woven – bugs and all.

Ahmed is an Azerbaijani artist who takes inspiration from traditional carpets made by craftsmen in his country. The artisanal carpets are made with intricate patterns and vibrant colors, both of which inspired Ahmed to begin working in textiles after years of focusing on paintings, video and installation art, Kate Sierzputowski writes for Colossal.

“Patterns and ornaments can be found in all cultures, sometimes similar, sometimes very different,” Ahmed tells Sierzputowski. “I consider them words and phrases that can be read and translated to a language we understand.”

Azerbaijani carpets are a prized around their world both for their beautiful patterns and the craftsmanship it takes to create such detailed, delicate pieces. The skills used to make the carpets are passed down through generations by family members, according to UNESCO. Traditionally, the carpets are dyed and woven in the winter by female household members who use special techniques to create intricate designs in the fabric. Carpets are often made to celebrate special occasions like weddings, a child’s birth and religious rites. Although the carpets carry much cultural significance, Ahmed takes pleasure pushing the craft's boundaries in his artwork.

More here.

The fear-mongers’ bad advice

Ken Roth in Politico:

ScreenHunter_1509 Nov. 24 13.46The horrendous Paris attacks have provided certain European and U.S. politicians with an irresistible opportunity to attempt to close the door on refugees while seeking to expand overbroad government surveillance. It is important, out of principle and for our collective safety, to reject these appeals to fear and prejudice, as some political leaders have done.

A fake Syrian passport reportedly left behind by an attacker of unknown identity and nationality suggests that he entered Europe with the recent refugee flow. The presence of the passport may reflect a deliberate effort by the so-called Islamic State to stigmatize the people who dare flee its “caliphate.” It has led to a chorus of voices seeking to keep out refugees and asylum-seekers, despite the fact that theattackers identified so far have turned out to be citizens of Belgium and France.

The focus on refugees in the aftermath of the recent attacks is a dangerous distraction from Europe’s violent home-grown extremism. The roots of the problem are notoriously complex but relate in part to the social exclusion of past generations of immigrants — the persistent discrimination, hopelessness, and despair that pervades neighborhoods on the outskirts of certain cities.

More here.

Tuesday Poem

The Grave, The Mine

Taking off from the city
at night, from the
airplane, I look at streetlights
below: hovering unfixed
sockets of light.

Then it is black beneath me.
A pair of headlights
veer slowly along a macadam
country road,
far from pianos and theaters.

Women are leaning back in taxis
Men stoop into taxis after them
and enter the well, the
grave, the tunnel, the mine
of fur and scent.

by Donald Hall
from Poetry Magazine