85 stories

Reykjavik Aurora

1 Share

Iceland Aurora Films have been busy filming the northern lights this winter, in rather unusual locations. This short film is all shot in the center of Reykjavik, Iceland and was extremely technically complicated to make due to the light pollution from street lights and houses. They also got really lucky with some incredibly strong Aurora displays this winter, and some Aurora shapes they had never seen before.  It’s beautiful.
Read the whole story
293 days ago
Share this story

SRE, CSE, and the safety boundary

1 Share

Site reliability engineering (SRE) and cognitive systems engineering (CSE) are two fields seeking the same goal: helping to design, build, and operate complex, software-intensive systems that stay up and running. They both worry about incidents and human workload, and they both reason about systems in terms of models. But their approaches are very different, and this post is about exploring one of those differences.

Caveat: I believe that you can’t really understand a field unless you either have direct working experience, or you have observed people doing work in the field. I’m not a site reliability engineer or a cognitive systems engineer, nor have I directly observed SREs or CSEs at work. This post is an outsider’s perspective on both of these fields. But I think it holds true to the philosophies that these approaches espouse publicly. Whether it corresponds to the actual day-to-day work of SREs and CSEs, I will leave to the judgment of the folks on the ground who actually do SRE or CSE work.

A bit of background

Site reliability engineering was popularized by Google, and continues to be strongly associated with the company. Google has published three O’Reilly books, the first one in 2016. I won’t say any more about the background of SRE here, but there are many other sources (including the Google books) for those who want to know more about the background.

Cognitive systems engineering is much older, tracing its roots back to the early eighties. If SRE is, as Ben Treynor described it what happens when you ask a software engineer to design an operations function, then CSE is what happens when you ask a psychologist how to prevent nuclear meltdowns.

CSE emerged in the wake of the Three Mile Island accident of 1979, where researchers were trying to make sense of how the accident happened. Before Three Mile Island, research on "human factors" aspects of work had focused on human physiology (for example, designing airplane cockpits), but after TMI the focused expanded to include cognitive aspects of work. The two researchers most closely associated with CSE, Erik Hollnagel and David Woods, were both trained as psychology researchers: their paper Cognitive Systems Engineering: New wine in new bottles marks the birth of the field (Thai Wood covered this paper in his excellent Resilience Roundup newsletter).

CSE has been applied in many different domains, but I think it would be unknown in the "tech" community were it not for the tireless efforts of John Allspaw to popularize the results of CSE research that has been done in the past four decades.

A useful metaphor: Rasmussen’s dynamic safety model

Jens Rasmussen was a Danish safety researcher whose work remains deeply influential in CSE. In 1997 he published a paper titled Risk management in a dynamic society: a modelling problem. This paper introduced the metaphor of the safety boundary, as illustrated in the following visual model, which I’ve reproduced from this paper:

Rasmussen viewed a safety-critical system as a point that moves inside of a space enclosed by three boundaries.

At the top right is what Rasmussen called the "boundary to economic failure". If the system crosses this boundary, then the system will fail due to poor economic performance. We know that if we try to work too quickly, we sacrifice safety. But we can’t work arbitrarily slowly to increase safety, because then we won’t get anything done. Management naturally puts pressure on the system to move away from this boundary.

At the bottom right is what Rasmussen called the "boundary of unacceptable work load". Management can apply pressure on the workforce to work both safely and quickly, but increasing safety and increasing productivity both require effort on behalf of practitioners, and there are limits to the amount of work that people can do. Practitioners naturally put pressure on the system to move away from this boundary.

At the left, the diagram has two boundaries. The outer boundary is what Rasmussen called the "boundary of functionally acceptable performance", what I’ll call the safety boundary. If the system crosses this boundary, an incident happens. We can never know exactly where this boundary is. The inner boundary is labelled "resulting perceived boundary of acceptable performance". That’s where we think the boundary is, and where we try to stay away from.

SRE vs CSE in context of the dynamic safety model

I find the dynamic safety model useful because I think it illustrates the difference in focus between SRE and CSE.

SRE focuses on two questions:

  1. How do we keep the system away from the safety boundary?
  2. What do we do once we’ve crossed the boundary?

To deal with the first question, SRE thinks about issues such as how to design systems and how to introduce changes safely. The second question is the realm of incident response.

CSE, on the other hand, focuses on the following questions:

  1. How will the system behave near the system boundary?
  2. How should we take this boundary behavior into account in our design?

CSE focuses on the space near the boundary, both to learn how work is actually done, and to inform how we should design tools to better support this work. In the words of Woods and Hollnagel:

> Discovery is aided by looking at situations that are near the margins of practice and when resource saturation is threatened (attention, workload, etc.). These are the circumstances when one can see how the system stretches to accommodate new demands, and the sources of resilience that usually bridge gaps. – Joint Cognitive Systems: Patterns in Cognitive Systems Engineering, p37

Fascinatingly, CSE has also identified common patterns of system behavior at the boundary that holds across multiple domains. But that will have to wait for a different post.

Reading more about CSE

I’m still a novice in the field of cognitive systems engineering. I’m actually using these posts to help learn through explaining the concepts to others.

The source I’ve found most useful so far is the book Joint Cognitive Systems: Patterns in Cognitive Systems Engineering , which is referenced in this post. If you prefer videos, Cook’s Lectures on the study of cognitive work is excellent.

I’ve also started a CSE reading list.

Read the whole story
906 days ago
Share this story

The forbidden zone, Alex Strohl

1 Comment

The forbidden zone, Alex Strohl

Read the whole story
947 days ago
Share this story

Display New Daily Cases of COVID-19 with Care

1 Share

Statistics are playing a major role during the COVID-19 pandemic. The ways that we collect, analyze, and report them, greatly influences the degree to which they inform a meaningful response. An article in the Investor’s Business Daily titled “Dow Jones Futures Jump As Virus Cases Slow; Why This Stock Market Rally Is More Dangerous Than The Coronavirus Market Crash” (April 6, 2020, by Ed Carson) brought this concern to mind when I read the following table of numbers and the accompanying commentary:

U.S. coronavirus cases jumped 25,316 on Sunday [April 5th] to 336,673, with new cases declining from Saturday’s record 34,196. It was the first drop since March 21.

The purpose of the Investor’s Business Daily article was to examine how the pandemic was affecting the stock market. After the decline in the number of reported new COVID-19 cases on Sunday, April 5th, on Monday, April 6, 2020, the stock market surged (Dow Jones gained 1,627.46 points, or 7.73%). This was perhaps a response to hope that the pandemic was easing. This brings a question to mind. Can we trust this apparent decline as a sign that the pandemic has turned the corner in the United States? I wish we could, but we dare not, for several reasons. The purpose of this blog post is not to critique the news article and certainly not to point out the inappropriateness of this data’s effects on the stock market, but merely to argue that we should not read too much into the daily ups and downs of newly reported COVID-19 case counts.

How accurate should we consider daily new case counts based on the date when those counts are recorded? Not at all accurate and of limited relevance. I’ll explain, but first let me show you the data displayed graphically. Because the article did not identify its data source, I chose to base the graph below on official CDC data, so the numbers are a little different. I also chose to begin the period with March 1st rather than 2nd, which seems more natural.

What feature most catches your eye? For most of us, I suspect, it is the steep increase in new cases on April 3rd, followed by a seemingly significant decline on April 4th and 5th.

A seemingly significant rise or fall in new cases on any single day, however, is not a clear sign that something significant has occurred. Most day-to-day volatility in reported new case counts is noise—it’s influenced by several factors other than actual new infections that developed. There is a great deal of difference between the actual number of new infections and the number of new infections that were reported as well as a significant difference between the date on which infections began and the date on which they were reported. We currently have no means to count the number of infections that occurred, and even if we tested everyone for the virus’s antibodies at some point, we would still have no way of knowing the date on which those infections began. Reported new COVID-19 cases is a proxy for the measure that concerns us.

Given the fact that reported new cases is probably the best proxy that’s currently available to us, we could remove much of the noise related to the specific date on which infections began by expressing new case counts as a moving average. A moving average would provide us with a better overview of the pandemic’s trajectory. Here’s the same data as above, this time expressed as a 5-day moving average. With a 5-day moving average the new case count for any particular day is averaged along with the four preceding days (i.e., five-days-worth of new case counts are averaged together), which smooths away most of the daily volatility.

While it still looks as if the new case count is beginning to increase at a lesser rate near the end of this period, this trend no longer appears as dramatic.

Daily volatility in reported new case counts is caused by many factors. We know that the number of new cases that are reported on any particular day do not accurately reflect the number of new infections. It’s likely that most people who have been infected have never been tested. Two prominent reasons for this are 1) the fact that most cases are mild to moderate and therefore never involve the medical intervention, and 2) the fact that many people who would like to be tested cannot because tests are still not readily available. Of those who are tested and found to have the virus, not all of those cases are recorded or, if recorded, are forwarded to an official national database. And finally, of those new cases that are recorded and do make it into an official national data base, the dates on which they are recorded are not the dates on which the infections actually occurred. Several factors determine the specific day on which cases are recorded, including the following:

  1. When the patient chooses or is able to visit a medical facility.
  2. The availability of medical staff to collect the sample. Staff might not be available on particular days.
  3. The availability of lab staff to perform the test. The sample might sit in a queue for days.
  4. The speed at which the test can be completed. Some tests can be completed in a single day and some take several days.
  5. When medical staff has the time to record the case.
  6. When medical staff gets around to forwarding the new case record to an official national database.

There’s a lot that must come together for a new case to be counted and to be counted on a particular day. As the pandemic continues, this challenge will likely increase because, as medical professionals become increasingly overtaxed, both delays in testing and errors in reporting the results will no doubt increase to a corresponding degree.

Now, back to my warning that we shouldn’t read too much into daily case counts as events are unfolding. Here’s the same daily values as before with one additional day, April 6th, included at the end.

Now what catches your eye. It’s different, isn’t it? As it turns out, by waiting one day we can see that reported new cases did not peek on April 3rd followed by a clear turnaround. New cases are still on the rise. Here’s the same data expressed as a 5-day moving average:

The trajectory is still heading upwards at the end of this period. We can all hope that expert projections that the curve will flatten out in the next few days will come to pass, but we should not draw that conclusion from the newly reported case count for any particular day. The statistical models that we’re using are just educated guesses based on approximate data. The true trajectory of this pandemic will only be known in retrospect, if ever, not in advance. Patience in interpreting the data will be rewarded with greater understanding, and ultimately, that will serve our needs better than hasty conclusions.

Read the whole story
949 days ago
Share this story

Racing thru the COVID-19 Fog: Maintain adaptive capacity in healthcare infrastructure and policy – Talk

1 Share

David D Woods (Cognitive Systems Engineering, The Ohio State University) David L Alderson (Naval Postgraduate School) Thomas P Seager)(Sustainability Conoscente Network & Arizona State University) […]

The post Racing thru the COVID-19 Fog: Maintain adaptive capacity in healthcare infrastructure and policy – Talk appeared first on Resilience Engineering Association.

Read the whole story
966 days ago
Share this story

VICTORY! New Free File rules ban tax-prep firms from hiding their offerings, allow IRS to compete with them (a love-letter to Propublica)


Six months ago, Propublica began beating the drum about "Free File," a bizarre, corrupt arrangement between the IRS and the country's largest tax-prep firms that ended up costing the poorest people in America millions and millions of dollars, every single year.

The scam is one of those baroque, ultimately boring and complex stories that generally dies in the public imagination despite its urgency, because "boring and urgent" is the place where the worst people can do the worst things with the least consequences.

With that warning, here's a short summary: in most wealthy countries, the tax authority fills out your tax return for you, using the information your employer already has to file every time it pays your wages. If all the numbers look right to you, you just sign the bottom of the form and send it back, without paying a tax preparer. If, on the other hand, you want to claim extra deductions, or if something complicated is going on with your finances, you can throw away that free tax return and fill in a form from scratch, either on your own or with the help of a professional.

When Americans asked to have the same courtesy extended to them -- a move that would save the vast majority of Americans millions and millions of dollars they were currently paying to the likes of HR Block and Intuit/Turbotax, every single year of their entire working lives -- the tax-prep industry mobilized to kill the proposal. The industry (which is highly concentrated and dominated by a small handful of firms whose top execs have mostly done time in all their competitors' board rooms, making them into essentially one giant company whose different divisions have different shareholders) lobbied the IRS very hard, and won a resounding victory.

That victory is called "Free File." Under Free File, each tax prep company is required to serve a slice of working Americans with free, online tax-preparation. The arrangement was hailed as a victory for public-private partnerships, harnessing the efficiency of the private sector to perform this public duty of the state. Importantly, it meant that the IRS would not expand its headcount or budget, both of which had been slashed by successive right-wing presidents and their legislative enablers. The move was cheered by anti-tax extremists like Grover Nordquist, who was delighted by the "efficiency" of you saving a bunch of pieces of paper the government already had, typing them into an online form, and hoping that a company's website came up with the same calculations that the government had already made about your tax-bill.

Part of the Free File deal banned the IRS from creating a competing offer and it banned the IRS from advertising the existence of the program or telling people where to find the free offering.

As soon as the ink was dry on Free File, the tax-prep companies set about to sabotage it. Intuit -- a massive company led by a bizarre cult figure -- and its competitors hid their Free File offerings deep in their sites, and used the "robots.txt" system to instruct search engines to hide them. They took out search ads for the phrase "Free File" that directed users to paid offerings with the word "free" in their names. They created "Free File" systems that would make you go through hours of work entering your data before surprising you with a notice that you didn't qualify for Free File because you'd paid interest on a student loan (or some other normal thing) and then ask you if you wanted to pay to keep your work and finish your tax-return in the non-free system.

There's a simple name for this kind of activity: fraud.

But it was a fraud in plain sight, one that went on for years and years, and which created a stealth tax on the majority of Americans, which they had to remit not to the IRS, but to the tax-prep companies, which used the money to lobby to make it even harder to get away from handing them your money every year.

Enter Propublica, whose relentless reporting did the seemingly impossible: it made a complicated, boring important thing into something that millions of Americans cared about. Something they cared about so deeply that they actually managed to shame the IRS into taking action.

Remember, the IRS is an administrative agency, under the direct control of the Trump administration. That means its commander-in-chief is a guy who said dodging his taxes means that he's "smart." While the IRS has many good, hardworking staffers, it has also been demoralized and gutted by the right, who have convinced millions of poor people that it's somehow in their interests if it's easier for rich people to duck their taxes.

Despite all this, the IRS has enacted new Free File rules: first, these rules ban tax-prep companies from hiding their Free File offerings, and it bans them from using deceptive names for non-Free File offerings (Turbotax will no longer be allowed to confuse Americans by offering "Turbotax Free" -- which is not free -- as a competitor to "Turbotax Free File," which is).

Second, the rule allows the IRS to develop its own competing Free File product, which means that the government agency that already knows how much tax you owe will allow you to review its findings each year and then either challenge them, or simply click OK, without paying a single cent of tax to Intuit or HR Block, and free you from filling in lengthy, bureaucratic forms.

This outcome is nothing short of miraculous: it did not come as the result of Congressional action. It did not come as the result of the Trump administration's inattention (the release came out the same day that the Trump administration revised its tax rules to allow money launderers to retain billions in the loot they've stashed offshore).

It came about as the result of fucking journalism. Propublica wrote its way into a better world, with relentless, deep, accessible reporting that made this boring, important thing come to life.

I am sympathetic to the idea that talking about politics isn't doing politics, but that's not entirely true. Learning about what's going on and telling the people you know about it and getting them to tell others is part of how we make change. Propublica's excellent reporting wouldn't have mattered if people hadn't read it -- and talked about it.

And Propublica has done this repeatedly over the past year, deeply reporting on naked, grotesque corruption in ways so vivid and undeniable that they actually changed things, and not in some abstract, boring way, but in ways that matter to the immediate, lived experience of real people who had been brutalized and poisoned and jailed and mistreated with impunity, for years, until Propublica wrote about it.

Here are some examples, just from the stories I paid attention to this year (Propublica does so much good work that I can't manage to cover all of it):

* Reformed South Carolina's "magistrate judge" system that let "judges" with no legal background and less training than barbers sentence poor people (most of them Black) to prison in defiance of their constitutional rights;

* Dismantled Illinois's system of Quiet Rooms where special ed kids were put into solitary confinement, sometimes for days at a time;

* Shamed a "Christian" hospital into ending its practice of suing thousands of patients, many of them its own employees, for inability to pay their medical debts, and forcing it to jettison the private army of debt collectors it kept on its payroll.

* Killed an Illinois scam whereby affluent parents temporarily gave up custody of their own children so they could steal college grants earmarked for poor children;

* Got two Louisiana cops fired for encouraging people to murder Alexandria Ocasio-Cortez;

In addition, Propublica has done lots of reporting that hasn't yet created political transformations, but has changed our debate and laid the groundwork for change to come: called attention to the penniless hero of the ransomware epidemic; discredited a "walking polygraph" system used by police forces to frame their preferred suspects with sheer junk science; documented the link between pharma company bribes and doctors' prescribing; named every former lobbyist in the Trump administration; tracked every penny of the 2008 bailout money; documented Wayne LaPierre's self-dealing from the NRA's war-chests; documented the grifty conservative PACs that scammed millions out of scared old white people with racist Obama conspiracies and then kept the money for themselves; published a blockbuster story on the theft of southern Black families' ancestral lands through a legal grift called "heirs' property"; debunked the "aggression detection" mics being installed in America's classrooms; outed a "ransomware consultant" that was working with ransomware crooks to simply pay the ransom, while pretending that they were able to get you your files back without enriching the crooks who locked them up; named and shamed Alabama sheriffs who lost their re-election bids and then spent thousands of public dollars on frisbees or stole discretionary funds, or destroyed food earmarked for prisoners, or drilled holes in all the department computers' hard-drives in a form of "vindictive hazing"; followed the payday lender industry to a Trump hotel where it staged an annual conference, funneling millions to the president's personal accounts shortly before Trump reversed Obama's curbs on predatory lending; documented how TSA body-scanners single out Black women for humiliating, discriminatory hair-searches; revealed the secret history of wealthy people destroying the IRS's Global High Wealth Unit; and did outstanding work on the Sackler family, a group of billionaire opioid barons whose products kickstarted the opioid epidemic that has now claimed more American lives than the Vietnam war.

2019 was a dumpster-fire of a year and 2020 could be worse -- or it could be the dawn that breaks after our darkest hour. Finding Propublica's victory lap on Free File on New Year's Day was just the sunrise I needed to give me hope for the year to come. Sometimes, simply finding the truth and telling it to the people can make a change.

I'm a Propublica donor, and an avid reader. I admit that sometimes when I see that PP has published another 15,000-word expose, I am slightly dismayed at the thought that I'm about to lose 1-2 hours of my life to digesting and writing up the new story, but that dismay is always overcome by excitement at the thought that they have turned over a new rock and found something genuinely awful beneath it, and that, with all our help, we can sterilize that foetid sludge with blazing sunshine.

The addendum to the deal, known as Free File, comes after ProPublica’s reporting this year on how the industry, led by TurboTax maker Intuit, has long misled taxpayers who are eligible to file for free into paying.

Under the nearly two-decade-old Free File deal, the industry agreed to make free versions of tax filing software available to lower- and middle-income Americans. In exchange, the IRS promised not to compete with the industry by creating its own online filing system. Many developed countries have such systems, allowing most citizens to file their taxes for free. The prohibition on the IRS creating its own system was the focus of years of lobbying by Intuit. The industry has seen such a system as an existential threat. Now, with the changes to the deal, the prohibition has been dropped.

The addendum also expressly bars the companies from “engaging in any practice” that would exclude their Free File offerings “from an organic internet search.” ProPublica reported in April that Intuit and H&R Block had added code to their Free File pages that hid them from Google and other search engines, diverting many users to the companies’ paid products.

IRS Reforms Free File Program, Drops Agreement Not to Compete With TurboTax [Justin Elliott and Paul Kiel/Propublica]

(Image: Phil Catterall, CC BY-SA, modified)

Read the whole story
1043 days ago
Share this story
Next Page of Stories