I recorded a year’s worth of Jon Brion’s earliest residencies and the first appearances by Tenacious D. Workshops of early material and intimate performances by Aimee Mann, eels, Grant Lee Buffalo, Glenn Tilbrook, and countless others. Stand-up performances by Margaret Cho, Janeane Garofolo, and Sarah Silverman. Michael Stipe and Helena Christensen singing karaoke.

And my personal favorite: Andy Prieboy.

https://andyprieboy.com/music-store

InternetArchive.NET is a type-safe library providing access to all features at Internet Archive.

Internet Archive’s API evolved over nearly two decades using a variety of technical conventions each fashionable at specific moments in time. XML? S3-like? REST-ish? JSON Patch-sorta? They got it. Many API remain in beta. Documentation is sparse. No fewer than seven different date formats are in use.

And it’s a fantastic resource.

The data source itself is an evolving schema implemented in PHP on the server typically consumed by curl or a Python client.

Using scripting languages on both the client and server is perhaps not the best approach to enforce consistency across 700 billion documents.

That said, I watched Microsoft slog through years of overengineered distributed systems technology and sat through endless meetings arguing about namespaces and identifiers. I’m not surprised that informal solutions like “if that doesn’t work, try two underscores” scales just fine to nearly a trillion objects.

Still, I couldn’t get anything done with the toolset as provided.

I reverse engineered the archive.org backend, discovered the expected quirks and security glitches, and provided the robustness and reliability necessary to treat archive.org like a true platform through careful engineering and custom retry logic.

InternetArchive.NET

John Leimseider (1952-2018) was a beloved and world-renowned electronic instrument technician. JL spoke frequently of his desire to create a free and open archive of technical information and documentation. With support from his family and a number of preservation organizations, we are pleased to make these materials available.

Remembering John Leimseider [nmc.ca]
John Leimseider Archive [archive.org]

Sharing so that the less-funded among us can enjoy it as entertainment, too.

From: Erik Gavriluk
Date: Mon, Jul 12, 2021 at 6:33 PM
Subject: La la la la – oh, what fun!
To: Rony Abovitz

Traditional hand tools set the standard for user experience. We use them to modify our environment and gather information about it. Being truly physics-based by virtue of existing in the non-virtual world, they provide feedback free from latency or quantization effects.

The tech industry continues to accelerate away from these principles. Even Apple suddenly found themselves unable to create a functional laptop keyboard. It took them four product generations to get back to baseline. Did they lose their way? Or did using their own tools lead them astray?

To explore, let’s go from Cupertino to Japan via Google, who credits Masaya Matsuura as a musician – complete with a photo of him in front of an ultra-rare analog synthesizer.

He’s better known as the digital inventor of the multi-billion dollar “press buttons in time to music” game genre, with multiple hit titles and at least one underground cultural marvel that recently got rediscovered and blasted all over TikTok.

But why did a 20 year old game with such primitive visuals suddenly explode in popularity all over the web? 

Character design? Maybe… once you get past the pixels. Music and voice effects? Fantastic but arcane. Game design? Um, the whole thing started as a quick one-off for a car commercial. 

It’s tempting to credit nostalgia but the people most excited about the game were born long after it was created.

I suggest instead that it’s yet another indicator of deep cultural awareness in the youngest generation. Unlike older people they don’t see the rabbit as retro so much as get a feeling that, whatever the heck it is, it probably shouldn’t look quite like that. They know something is going on, but more importantly they know something is missing. And they’re hungry for it.

In fewer than 30 lines of code I made this. And now young and old alike immediately get it – and feel it.

Now let’s travel from the Sony Playstation offices in Tokyo down to Shizuoka. In 1991, Mr. Suzuki bought all the 1930s tooling needed to build Hammond Organs. Today Hammond-Suzuki is the sole manufacturer of electronic music keyboards that have proper key travel. Everything else is awful and, just like the Apple keyboard mess, musicians complain about feel and, yes, even keys that play notes multiple times. Yet this music creation equipment costs thousands of dollars more than even Apple products. We have lost our way.

Above, the mechanism invented by what was then the Hammond Clock Company reveals a series of contact switches that connect nine busses to a crosspoint of electromagnetic pickups located near 91 precision gears ground at the clock factory. Each gear is locked to a capstan driven by a synchronous motor spinning at the 60 Hz electrical line frequency. 

When pressing a key, the performer engages these 9 switches gradually, changing the attack and tone of the note. The mechanism also generates a distinctive “keyclick” sound and, unsurprisingly, no one is happy with the digital versions. Makes sense when you have a peek through the plexiglass (itself invented in the 1930s).

Later, different gear ratios were needed for 50 Hz countries. As a result, emulations of classic 1940s American Jazz equipment digitized by 1990s Eurodance software companies can sound a bit disconcerting to people who grew up listening to R&B records – regardless of where they grew up or what voltage comes out of their wall.

Pianos are a relatively recent phenomenon – the organ precedes it by at least six centuries. At this point of the diatribe you might imagine how things worked out when, back in 1983, technical people invited themselves to put a layer between musicians and computers and create a Musical Instrument to Digital Interface.

The engineers behind MIDI oversimplified everything to “note on” and “note off.” Because digital. Being enthusiasts they couldn’t resist adding stuff like “after-touch” and other unnecessary abstractions. Yet they couldn’t come up with a bi-directional protocol. As a result, actual PANIC buttons were placed on instruments to stop commonplace cacophonies of beep-bloops from assaulting audiences when things fell out of sync.

A slow serial link was used, inspiring such technical innovations as “groove quantization” whereby a 1980s Atari would attempt to add “groove” back to your performance after losing the nuances during capture and playback due to latency. 

Finally, values didn’t even use all 8 bits available. They threw away half the range during quantization so 8 bit computers wouldn’t have to worry about those pesky signed operations. So 7 bits is all you get, rock on.

Technologists are often drawn to music but seldom know what to do once they get there. Follow the path of recorded media: from three-dimensional wax cylinders to flat circles (78 RPM) to smaller, slower discs (45 RPM) then finally “long playing” albums (33 1/3 RPM). Or with magnetics: from 2″ wide tape traveling on precision guides at 30 inches per second to 4 tracks spread across 0.15″ tape wobbling in the dashboard of your car at 6% of the rate it was originally recorded.

This isn’t just bean counting and mass production: everyone from Ray Dolby to the LearJet guy grafted on technology and celebrated it as genuine advancement. I think they even believed it. But while learning what magnetic tape actually does on a hedonic level we found ourselves heading in the opposite direction. And even crude formats like LPs and Cassettes are making a comeback versus what we have now. Curious.

It’s not nostalgia! There’s a fundamental impedance mismatch between the digital stuff we’re handed and what people actually need and crave. Musicians, filmmakers, gamers – they’re all trying to tell us something.

Forget the great industrialists – even the modest ones were not afraid to build factories, let alone tools. Hammond made clocks, robotic card dealing tables, and musical instruments then gave music performances in the Ford Pavilion at the 1939 World’s Fair.

The Minnesota Mining and Manufacturing company started in 1902 with a goal of mining abrasives to manufacture sandpaper. Failing at mining, they bought the abrasives instead. The floor of the factory collapsed under the weight of the first raw goods delivery. Two decades later they noticed people using their sandpaper at the Ford factory were having trouble painting cars. So they invented masking tape. 

Then they stuck magnetic particles on tape and designed what many regard as the best recording equipment ever, probably because many of the people working on the project were also musicians. Later, the 3M Orchestra (!) of over 100 employees recorded instrumentals for the Cantata 700 – the largest audio cassette ever produced holding 3 times the number of songs of the original iPod. It quietly generated profits for decades. 

This isn’t Bell Labs level stuff. This model is totally reproducible by mere mortals in 2021. C’mon, they had a friggin’ board game division!

Laurens Hammond: “A good field for the inventor is toys for adults.” Cool, let’s make it fun.

Exactly 100 years later and here we are: terrible input devices that actually injure people. $1000+ handhelds with 450 ppi screens yet somehow we still see pixels. Basic signal processing and even gamma that isn’t handled correctly from browsers and games up to high-end A/V software. $300 round thermostats with 2.5 star reviews right next to the $19 one we all grew up with 5 stars. $500 proprietary headphones that can’t receive CD quality digital audio. A $100 digital pencil soon to be in its third tech generation. And still working on that 3D glasses thing.

What’s missing? A little bit of that…

Analog Magic (Inc.)

The problem: digital computers were invented to perform accounting and other numerical operations. If instead they were designed to support creative tasks they would be quite different in structure. (Dan Sandin, 1973.)

We are going to fix this. Input devices. Output devices. Processing tools in the middle. Automated assistants that actually provide assistance. Cloud services that actually provide service.

Why? Nobody wants to go back to work yet nobody wants to go to the movies, either.

“Platforms are collapsing. Distribution is as unbounded as it is broken. The market for consumption is larger than ever.”

You’ve heard that all before. But somebody has to make the stuff and tooling either doesn’t exist or is hopelessly stuck in the 1980s. Yes, even the Apple stuff.

Where? Primarily Atlanta, GA; intermittently Los Angeles, CA; hopefully Wellington, NZ.

Who is the target market? Everyone. If you make stuff, which everyone does, we’re going to make making it better.

No really, what is the market? Unbounded, because all tech that interfaces to the real world is broken. (We’re not surprised that Google Voice never got around to detecting hangup calls and instead emails you a loud click with a blank text transcription then lights up your phone with a notification. But 15 years later when Twilio goes public and hits $60 billion and does the same thing? C’mon already.)

Is this a media company? No. Media is interesting but we’re a maker-for-makers company and makers ourselves. We build tools. Then we build tools with those tools. Repeat.

Is this a music or video or “art” company? No. Musicians and filmmakers and artists are interesting but just like every company was forced to become a tech company to survive, now every person is forced to become a media producer just to survive on social media and video calls.

How do you constrain this?

We derive product inspiration from time-proven tools that already exist in the real world.

• You don’t need to read Dieter Rams (or nick his designs) if you remain physics-based and reality-based.
• Building tech products based on historically proven concepts and existing physical designs greatly accelerates time to market (30 products in 3.5 years in a previous company).
• Unlike the competition, we do not waste time ornamenting our tools with purple unicorns, red antelope, or pink flamingos.

We do not create products that require latency in order to function.

• Either the task is performed below the range of human perception or to an existing analog standard.
• Anything else simply destroys the product or experience, no exceptions.
• This restriction alone eliminates an enormous amount of screwing around in R&D.

We solve problems in the proper domain.

• An analog to digital conversion, a Fourier transform, three lines of code, an inverse Fourier transform, and a digital to analog conversion is one way to solve the problem.
• But a three cent resistor and a three cent capacitor can eliminate both a three dollar microcontroller and three person-years worth of software.
• Machine learning and frequency domain for analysis and open-loop feedback only; generation or resynthesis via systems based on traditional analog or physics-based models.

We build systems that work for humans.

• A useful tool performs a task and never interrupts the user from that task.
• A tool or system doesn’t need to disrupt an entire industry to be considered useful or worthwhile.
• Jidoka: “Automation with a human touch.” 

Are you building a platform? A non-goal and a greatly misunderstood concept in 2021. But given our systems expertise it wouldn’t surprise us if one arose organically.

Are you building tools or are you building services? Yes.

What’s the craziest thing you want to build? Besides the card game division? Random scan displays.

Who needs that? Everyone, but games alone justifies R&D.

Do you think there’s a market? Nvidia has five different names for technologies hacking at the problem and none of them works.

Do you need to build that to be successful? No, but it would be great fun.

Can you give me an example of an actual product? Tape-based analog audio reproduction is mathematical differentiation over magnetic flux. We analyzed dozens of classic recording machines and tape formulations at the raw physics level. Then built models and developed programs to replicate all aspects of the system.

Do you need a super fast Nvidia GPU or cloud service to run something like that? No, in fact it doesn’t even require a digital computer. As such, it replicates the original system free from latency or quantization effects.

Can you give me an example of a more traditional product?

After record companies finally found the cheapest platter still acceptable to consumers, the final insult was doubling the bandwidth required in order to support Stereo. Two ears, two speakers – what could possibly go wrong? Well, one groove.

Forcing Stereo into the existing technology was so complicated that an entire sub-industry formed around the problem. None other than military giant Fairchild got into the action, building a 100 pound tool containing 11 transformers and 20 vacuum tubes. A new class of “mastering engineer” was required to operate it: controlling a postwar German cutting lathe while simultaneously twisting Fairchild knobs to limit the lateral and vertical forces that would throw the needle out of the groove.

During the golden age of music studios, taking an impeccably-recorded analog master and doing the least damage possible transferring it to a reduced-bandwidth medium seems like a feasible task for a trained engineer plus a tool. Especially when that tool is built by an aerospace company (grounded in physical concepts like gravity) with bonus expertise developing analog computers for gun sights as a wartime side gig.

But 60 years later the notion that a human can do this on music recorded by millions of amateurs in environments ranging from bedrooms to the finest studios and somehow make it sound better to everyone, everywhere listening on everything from CDs to Spotify streams to laptop speakers to earbuds to YouTube? 

Well, that seems like a bit of a stretch. Some additional tooling might be useful.

How long would it take to build something like that? All done. But there’s a DJ in Orange County who thinks it could use a little more bass, so I suppose these things are never quite finished.

Dr. Meredith certainly didn’t lack credentials: honors graduate from Stanford, Ph.D. from Brandeis, Harvard fellow, taught at MIT.

Perhaps more impressively, he convinced Hot CoCo not only to publish pages upon pages of 6809 assembly language, but also to provide additional space so that he could explain the concepts in a way that I could understand. As a twelve year old, it served as my introduction to assembly language and computer graphics.

Professor Meredith was a great teacher, confident but humble. Which reveals this sentence in his article not as bravado, but a call to action:

I do not believe this program could be recoded to run more than twice as fast as it does now.

Challenge accepted! Source code and a disk image:
https://github.com/milliluk/display

Thanks to Russell Lindberg for preserving the original source material necessary to complete this project. And thank you, Professor Meredith, for a career in tech and the arts.

Acting as an informal advisor because they never got the paperwork together.

From: Jake Lodwick
Date: Tue, Feb 12, 2019 at 7:01 PM
To: Erik Gavriluk

Tracker v1 is nearly ready to go. Just gonna QA with Francis and then submit to Apple so Ye can easily install it.

I would love to know your prediction for what happens in the first week of launch!

From: Erik Gavriluk
Date: Tue, Feb 12, 2019 at 7:08 PM
To: Jake Lodwick

It is not a music product, it is not a consumer product, consumers are exhausted of apps and musicians are exhausted of Kanye.

I lean toward little to no traction with the best case scenario being minimal brand damage.

It is certainly a very cool feather in your cap but maybe you’re too old to be wearing caps.

From: Jake Lodwick
Date: Wed, Feb 13, 2019 at 1:03 PM
To: Erik Gavriluk

Thank you! All predictions have value. 

From: Erik Gavriluk
Date: Fri, Aug 25, 2017 at 3:08 PM
Subject: Big Knobs. Big Sound.
To: John Lagomarsino

Loved your article!

The reason for the audio software industry’s knob obsession? Me.

Bomb Factory launched in 1999. We were the first audio company to license brand names and work with original equipment inventors. Before that, companies didn’t dare say “Hammond Organ.” Mega-corporations like Roland would spend millions of dollars designing a keyboard then slap a “Ham and Organ” preset on it.

Instead we worked directly with original inventors like Dr. Robert Moog. We lifted them into the digital age. In return they kept us honest on authenticity and quality.

We saturated five different magazines with full-page, full-color ads for six months. Then we shipped. And our competitors immediately dropped to near zero revenue.

But something else happened, too. Timed to our launch, a hacker released an industry-wide crack tool. So all companies (including us) actually went to near zero revenue for a while.

Competitors overreacted and thought we’d eaten their lunch. They decided to do what we did. And they’re still copying designs I made in the ’90s, only worse.

My Pultec (note the minimal amount of ornamental lights and screws):

Just a few of the copies that shipped after:

What was supposed to be a very specific thing tied to a very specific type of professional audio software somehow became the standard for most audio software. “Big Knobs, Big Sound” flyer from decades ago, attached.

Pro Audio, despite the glamour, is an industry with very little money, very little expertise, and very little innovation. In what other industry could a high school diploma and being Billy Joel’s keyboard tech qualify you to be a CTO?

That said, the whole “anti-skeuomorphism” movement is a bit of an overreaction. In discussions with high level people at Apple I never spoke about skeuomorphism. We spoke about affordances — and skeuomorphism is an over-used word that represents just one trick in that bag. The problem is less the skeuomorphism and more the fact that most people (even Apple) suck at it.

Cluttering up a computer screen with non-functional elements like handles and missing screws is pure idiocy. But hey, they had to make theirs look different somehow.

From: John Lagomarsino
Date: Wed, Aug 30, 2017 at 11:17 AM
Subject: Re: Big Knobs. Big Sound.
To: Erik Gavriluk

Hi Erik,

This is FASCINATING stuff, and I’m actually getting emails from people that are painting a pretty clear “how did we get here” picture. It’s totally mind-boggling that Propellerhead, for example, still makes Reason look like a rack, but their mobile UIs are so clean and intuitive and fun.

I’m considering doing a follow-up piece, maybe in podcast form, that’s a little more reported out. I think it might make a good segment if I talk to a few people involved in this industry about it, and how interfaces haven’t offered enough affordances as the industry has grown up.

Thanks for reaching out. It really makes so much sense how this all happened, given your backstory.

As teenagers in the 1980s, Greg Miller and I wrote countless lines of assembly-language code for the Motorola 6809 microprocessor. Often considered the best (certainly the most fun) of the 8-bit micros, it failed to achieve the widespread success of the MOS 6502 and Zilog Z80.

The 6809 arrived at an awkward time for both manufacturing process and the industry as a whole. Motorola couldn’t get the price down and had a next-generation 16 bit processor waiting in the wings.

Many theories exist for the 6809’s underperformance in the marketplace. Cost was an issue but nobody could get their head around the thing, including Motorola themselves.

They sure tried. Motorola not only made the 6809 source code compatible with other chips, they supported relocatable object code, hoping to sell ROMs full of prefabricated subroutines.

Fifty years later, the 6809 still seems to have just a few more features than people can comfortably fit into their head. Open source projects, emulators, even basic tutorials frequently stall around 90% complete. There are at least ten emulators and a half dozen operating systems for this modest chip yet I couldn’t find a modern cross assembler that could generate accurate binaries from 1980s source. One actually took longer to assemble code on a modern machine than a vintage one took to assemble code off floppy disks in 1987.

After submitting patches to an open source assembler, making modifications to the 6809 core in MAME, and releasing source code to an app that Greg Miller and I wrote in the 1980s, I noticed that the retro-computing crowd was moving toward FPGA emulation.

Being able to assemble code accurately doesn’t mean much if nobody can run it. And despite the 6809 being used in several of the top video games of all time (a common motivator in the emulation crowd) nobody was stepping up.

So Greg and I decided to get the band back together.

The result is a pin and cycle accurate 6809 and 6809E CPU core built from the data sheets up. Verified with a logic analyzer, exhaustive tests against known good emulators, and via circuit compatibility in the actual CPU socket of vintage computers and arcade games. (The validation page is a fun read.)

Greg did the heavy lifting on this one and I served mostly as project manager. Although I did get to write a patch for a decades-old glitch in Williams’ Stargate.

Released as open source with a permissive license, it’s the standard core now used in emulators and even industrial applications.

https://github.com/cavnex/mc6809

When you’ve got twenty minutes and really want to feel something, read this:
The Sixth Stage of Grief Is Retro-computing (Paul Ford, 2014)

From: Erik Gavriluk
Date: Thu, Nov 5, 2015 at 9:21 AM
Subject: Re: OneDrive
To: Bob Muglia

It would be neat to get paid for these sorts of insights.

From: Erik Gavriluk
Date: Fri, Jun 26, 2015 at 1:47 AM
Subject: Re: OneDrive
To: Dick, Joe Belfiore

Just throttle it all way back. Pull back the marketing. Slow down the rollout of “unlimited” storage. Seriously… who greenlit that? 

From: Erik Gavriluk <[email protected]>
Date: Sat, Jun 20, 2015 at 12:16 PM
Subject: OneDrive
To: Joe Belfiore

Heads up on OneDrive — this thing is really not good. Obviously one person’s anecdata isn’t a useful survey but I’ve seen numerous behaviors that reveal a fundamental lack of engineering process. It fails in common scenarios and breaks spectacularly in slightly-uncommon ones like copying from a Unix-based file system.

I’m sure you’re swamped but this is the kind of glitch that could derail what otherwise appears to be an amazing upcoming release of Windows.

I don’t cry wolf and I feel obligated to sound the alarm. Everyone I know is having problems with this damn thing. It’s really bad.

Is there someone I can talk to?

From: Joe Belfiore
Date: Mon, Jun 22, 2015 at 2:31 AM
To: Erik Gavriluk
Cc: Dick

Thanks for sending.

Dick on the CC line runs program management for OneDrive, and he’s interested in hearing more specifics. Getting sync reliability (healthy clients, effective service-side scaling, etc. etc.) has been a big priority for the team and they are rolling out a bunch of work in that dimension.  Your timing to ask is good.

From: Erik Gavriluk
Date: Fri, Jun 26, 2015 at 1:47 AM
To: Dick, Joe Belfiore

Hi Dick,

I spent the past few days contemplating a response. I would like to commend your team for reaching out to me and particularly their positive attitude in doing so. It says a lot. I pruned my response back to just you and Joe due to tone, which I apologize for in advance.

Sync is an odd blind spot for Microsoft. I swear I never had a Microsoft product sync correctly. Calendars in Outlook 97, files on the PocketPC, contacts in Windows Phone 5 and 6 (and 6.1, and 6.5…), songs on Zune, saves on Xbox, my tax returns on Windows Home Server, entire databases via Azure SQL Sync.

I can barely list all the platforms I’ve lost data on let alone the numerous times I’ve lost data on each platform. FTP, WebDAV, LDAP, IMAP — and hey, remember that Live Mesh disaster? — if there’s a way to copy data over it, Microsoft’s found a way to mess mine up.

But hey, it’s gotta be me, right?

DropBox. Mozy. Copy. Time Machine. All coded in half-assed languages by second-tier software companies. Never had a problem with any of ’em! The only time I see a popup is when they want me to give them money. Which I do, happily, because their stuff just works.

Windows Backup? Just popped a notification while I was typing this saying it couldn’t find the server. Meanwhile Mozy’s changed corporate ownership twice since I’ve been a subscriber yet it just keeps working.

Just glanced down at the notification bar and now I have two OneDrive notification icons for some reason. (Hovered over one and it silently disappeared.)

I simply do not understand.

Sync is hard! I certainly don’t have to tell you that. But it’s containable! Unless…

Prior to Windows 8.1, we had two sync experiences. One used on Windows 7/8/Mac to connect to the consumer service, and a second sync engine to connect to the commercial service (OneDrive for Business). In Windows 8.1 we introduced a third sync engine…

https://www.microsoft.com/en-us/microsoft-365/blog/2015/01/07/taking-the-next-step-in-sync-for-onedrive/

Oh, good lord.

I don’t know what’s possible to fix ahead of the Windows 10 launch but something has to be done with status and error reporting at a minimum. This is critical. I’ve hand-soldered prototype SCSI interfaces yet I cannot build a mental model of what OneDrive thinks it’s doing while copying data. I saw OLE-style error codes! They had cobwebs on them! (Tip to senior managers: if you see anything like 0x8000a0a3 in end-user facing dialogs that means “Too many technology layers. Please simplify.”)

As best as I understand it — admittedly I’m on Windows 7 and I’m a consumer so I only have a subset of “sync experiences” — OneDrive seems to consist of the following:

• A slow, buggy file scanning component that uses excessive RAM
• A headless notification component that seems to miss local file system notifications yet overreact wildly to remote network notifications
• A glitchy bandwidth throttler that varies between “do nothing at all” and “send so much data it crashes my Airport Extreme” — requiring a power cycle
• A user interface component straight out of the Windows 3.0 design era that reports strange errors but gives you no help in solving them. (And, after you manually fix the errors, continues to display them.)
• A system tray widget that only knows how to say “Scanning for changes…” and launches the old “skydrive.live.com” URL when I click on it
• Plus a web-based client and some backend indexing tech that is truly brilliant and best of breed which is why I’m even bothering to go through this exercise

Dropbox is presently using 125MB of RAM. This is excessive and wrong and always bothered me. OneDrive is presently using 860MB and has been burning 20% or more CPU since mid May. 

[ Update: my sync completed! Now OneDrive is at 25% CPU utilization and using nearly two gigs of RAM. ]

The bugs I’m seeing indicate fundamental design flaws and sloppy engineering. One of your developers mentioned “adding more telemetry.” It’s summer 2015! It should’ve had complete telemetry half a decade ago before it even copied the first byte.

I realize this is not your fault. And yes, we both know that this technology stack has been a career-ender for nearly every person who comes near it. Sync is just something that, for whatever reason, seems incompatible with Microsoft’s method of specifying and building technology. (I have pages there, too, but now’s not the time.)

To wrap this up as briefly as I can: I’ve seen bizarre, bug-report-defying sync behavior on Windows 7, Windows 8 and Mac. I’ve seen it on OneDrive Business and OneDrive: The Consumer Experience (now in 3D IMAX!) I’ve seen it in secure corporate environments with locked down hardware and networks (so I know it’s not malware or my config).

Obviously I’m an outsider and I don’t know what the plan is around Windows 10 but I don’t see any value in risking it on a OneDrive Hail Mary. Just ratchet it all back. 

Clean up status indicators, bandage up the error reporting, and pull everything back to conservative defaults. Copying speeds, caching, retries, RAM utilization, hell — even allowable storage space. Just throttle it all way back. Pull back the marketing. Knock it down two slots in the Save dialogs. Slow down the rollout of “unlimited” storage. Seriously… who greenlit that?

There are no cloud storage wars. Nobody gives a damn about Google Drive. Box? Dead, and Sinofsky’s screwing around over there, so you know it’s double dead. Dropbox is not a real company and we all know it. So let’s not screw up the Windows and Office franchises for the sake of some pretend space race.

From: Erik Gavriluk
Date: Fri, Jun 26, 2015 at 1:33 PM
To: Dick, Adam, Steven, Joe Belfiore

Ha, yes, Exchange was a bright spot! I still couldn’t get my Gmail copied over to Outlook.com though, and maybe that’s more important in 2015. Happy to provide more actionable details. Here’s some of the nuts-and-bolts stuff I pruned from my previous message. Please ignore the tone. Adam and Steve, ask away!

I have two Dropbox accounts, both contain just under 1 TB of data. OneDrive’s shared folder functionality limitation is embarrassing (how did that happen?!) but that’s clearly getting fixed so I figured I’d start migrating.

1.8TB of data on a 4TB drive. Copy everything from D:\Dropbox to D:\OneDrive. How hard can it be?

It took me over seven hours to fix all the “path too long” and “invalid characters in filename” problems. And I’m sure I screwed up some project files in the process. (There was even “robocopy” involved — that thing is maximally stupid, too.) I had to boot a Linux shell to speed up the process. This is madness.

The user interface couldn’t have been worse. It would show a few errors and give me no way to launch the problem folders. Then I’d manually fix and it would start scanning again… sometimes. Minimum of 30 minutes before the next batch of errors appear. And repeat.

Ergonomics aside the technical limitation is simply unacceptable. We got long filenames working on DOS… surely you guys can figure this out. 

My files are on the disk! Twice! They’re already synced in DropBox! You simply have to superset the DropBox and filesystem functionality. There is no other option. There is no debate. “Restricted characters?” No! The files are already on my disk! How can you ship a cloud service in 2015 that has less functionality than a DOS filesystem?

Anyway, that battle complete, I looked at my OneDrive tray icon, it said “Up to date.” Dragged the files in, and… “Up to date.” Waited a half hour. “Up to date.” Rebooted, went to bed, the next morning it was “Scanning for changes” again.

That was Wednesday June 17. The PC is on 24/7. I don’t use it very much. It’s on a gigabit switch. Speedtest.net just informed me of a 77Mbps upload rate and 11ms ping time. As of Friday at 8pm it had not yet so much as created the destination folder on the server.

On Saturday it finally started copying files. I pulled up onedrive.live.com on my Mac and saw the folder, finally. Saw a picture I liked, accidentally clicked “delete” instead of “download” in the web interface, then clicked “undo.”

The PC immediately went from copying to “Scanning for changes…” and sat there doing nothing for 24 hours.

145,000 files. 980 GB. “Properties” on the Dropbox and/or OneDrive folder gives me these stats in under a minute. 

What the hell is it doing? Is it hashing? Thumbnailing? Text indexing? Chatting with the server? Do I have too many files? Does it not like some of my media files? 

Couldn’t tell you, because there’s no status. No progress bar. No “Estimated completion: July 7th.”

It’s all so primitive and weird and half-assed.

[ Update: once it started, it copied 995.39GB in under five days. I consider this spectacular. ]

Prior to May, my experience with OneDrive was Mac only. The thing wasn’t even close to working on MacOS until the most recent updates. I’m sure you know this; I’ll spare you the details. But it was… bad. Really bad. Unprofessional, brand-killing bad. Even had an icon that drew incorrectly on the title bar. Multiple updates, but still with the wonky icon. That stuff is important. 

Ever watch someone close a big sale with bad product but really white teeth? If nothing else, at least get the teeth right. And hey, just snapped a grab of the dual notification icons on Windows:

From: Erik Gavriluk
Date: Sat, Jun 27, 2015 at 2:15 AM
To: Adam, Dick, Steven, Joe Belfiore

Now y’all are just pranking me.

From: Erik Gavriluk
Date: Sat, Jun 27, 2015 at 2:30 PM
To: Adam, Dick, Steven, Joe Belfiore

Thanks for the awesome reply, Adam. I’ll send logs and tech notes to you directly.

I would like to make a gentle caution about prioritizing program management and engineering around things like “0.14% of our active users hit this error…” though.

There’s a guy named W. Edwards Deming and everyone on this email should read him. If OneDrive were rock stable then you can sort data like that. But in OneDrive’s current state you need to focus on WHY 0.14% of the people are having that problem (hint: the answer isn’t buffer sizes) and address the problem holistically. This actually explains why it’s the #3 biggest feedback issue given the tiny failure rate. The natural inclination is to ignore that fuzzy number and focus on the hard data. But even a quick skim of Deming’s Wikipedia page will convince you that’s not the way to go.

I’ll beat this drum for one more paragraph because it’s important: I can’t tell you how much hard data there was around the first Xbox controller. Everybody internally knew it was awful, but one guy had all the data, and, well, now they teach it as an example of what not to do in business classes. Sinofsky endorsed an insane 6,000 page blog post talking about what people do in Windows Explorer. In the hard data “Open a File” didn’t even make the top 10. Four years later he’s still typing long posts — but now on LinkedIn trying to convince people he didn’t single-handedly bring an entire industry to a screeching halt. The resulting interface was such a disaster that Dell now points their screens away from consumers, and prefers to show empty wallpaper rather than even hint that their machines run Windows. Why? Because people still can’t figure out how to open their shit.

Don’t be a Sinofsky.

And don’t let my nonsense bring you down! The browser portion of OneDrive is already amazing. The transfer and storage component makes the competition look like toys. There are 50+ companies working on some idiotic subset of this in music, photos, legal documents — basically every MIME type gets its own startup that pretends to solve this problem. Once you guys get it stable you’re not going to believe what people do with it as a platform.

You will win and only Microsoft knows how to win that side of the game.

From: Adam
Date: Sun, Jun 28, 2015 at 1:37 PM
To: Erik Gavriluk
Cc: Dick, Steven, Joe Belfiore

Thank you Erik for turning me on to W. Edwards Deming – I was blown away by even a quick read of Wikipedia by how applicable his ideas are to the problem of sync, where the instantaneous state of the quality of the system is inherently unknowable.  Made me think of some mistakes we have made in the past – trying to get perfect and truly comprehensive data on one slice of the problem and miss entire sections of the experience.  Result is we don’t really measure quality correctly.  Anyway there is lots here and I am hungry to learn more.  Wish I knew of him earlier.  Just ordered his 1993 book.