[otn / people / robbat2 / resumé]

Resumé

(Printable version)
Name: Robin Hugh Johnson
E-mail: robbat2(0)orbis-terrarum.net [replace (0) with @]
Present Address: Gibsons, British Columbia, Canada

Looking for

  • Interesting remote-only opportunities around the level of new staff engineer (remote-only since 2006, no relocation, travel possible).
  • My recent experience is around cloud-scale Object Storage systems and microservices to enable Object & Block Storage (Ceph), including deep introspection into metrics and production engineering & SRE of Storage.
  • I'm open to things that are not Storage related. If you have a role that's looking for the ability to dive deep into brand new material, while still maintaining a broad knowledge base, please reach out to me!
  • Utilizing my 25 years of open source experience to improve your company.

Experience

  • DigitalOcean
    • Senior Engineer II (Storage Edge). 2021/March - 2023/Feb.
    • Senior Engineer I (Spaces). 2018/June - 2021/March.
    • Developed, deployed & maintained Ceph RADOS Gateway (RGW) and other storage services, with extensive integration work to offer Object storage as a public cloud service (exceeding 50PB published capacity), including work on HAProxy, Envoy, L4LB and custom microservices in Lua & Golang
    • Deep knowledge of S3 specification vs implementation nuances (see Published Papers & Conference Talks below); provided critical knowledge about undocumented/nuanced AWS S3 behaviors to develop an alternate S3 gateway.
    • 2019-2021: Provided extensive improvements to Ceph RGW metrics via Prometheus and capacity forecasting. Analysis of capacity data to find specific underutilized capacity, and implement operational improvements to recover that capacity (work later developed into new functions of dedicated teams).
    • 2019-2021: Examined company-wide time synchronisation issues beyond NTP: Identified problems from unrecognized dependencies and lack of service ownership for critical components. Helped negotiate new service ownership for components
    • 2020-2023: Developed numerous novel security measures to detect, validate, preserve, block and/or remove content with security/abuse content. e.g. detect/block feedback loops around L7-specific DDoS attempts at scale.
    • 2018-2022: Represented DigitalOcean on the Ceph Foundation board
    • 2023: Enabled team to accelerate feature work by developing & leading brand new Production Engineering role.
  • DreamHost DreamObjects. DreamObjects/Senior Cloud Storage developer. 2015/August - 2018/May. Developed & deployed Ceph RADOS Gateway (RGW) for a production 4PB Ceph cluster (with live data older than Argonaut release) dedicated to S3 and Swift usage. Ceph migration planning & implementation from legacy Ceph to new clusters. Developed & deployed additional S3 functionality via HAProxy Lua interface. Web-scale network interoperation via Cumulus Linux & Quagga/BGP.
  • Gentoo Linux (open-source volunteer work). Various positions:
    • Senior Developer. March 2003 - present. Responsible for development of base-system/openrc (focus on oldnet/netifrc); packaged Git, OpenLDAP & numerous other applications (formerly also MySQL, PHP, Courier-IMAP & Qmail).
    • Infrastructure Team
      Lead: 2007 - present.
      System Administrator: 2004 - present.
      Administration of the numerous servers as well as CVS, SVN, Git, LDAP & Bugzilla et al. services for the distribution. Managed system administration team of 8 staff.
    • See also Gentoo Foundation and Gentoo Council listed chronologically below.
  • Gentoo Foundation as non-profit/volunteer. Responsible for business and financial decisions for the Gentoo Linux distribution. Performed forensic accounting reconstruction and worked on IRS 990 filings.
    • 2009-2011 term: Elected trustee
    • 2011-2013 term: Elected trustee
    • 2015-2017 term: Elected trustee, treasurer
    • 2017/07/01-2017/12/23, assistant treasurer (trained new treasurer)
    • 2017/12/23-2018: Treasurer
    • 2018-2020 term: Elected trustee, treasurer
    • 2019-2021 term: Elected trustee, treasurer
    • 2021-2023 term: Elected trustee, treasurer
  • Orbis-Terrarum Networks Inc. 2005-present, incorporated in 2017. Own small business (iteration of non-incorporated entity further below). Directed & managed business to provide distributed cloud storage development & operations. Contract development customers include DreamHost, implemented RGW S3 static website functionality for their DreamObjects platform.
  • BC Libraries Cooperative/SITKA. Senior System Administrator & Architect. January 2012 - July 2015.
    Revamped software deployment, managed hardware refresh, provided support and maintenance for the open-source Evergreen ILS system (based on Perl & PostgreSQL);Migrated hardware systems to Ganeti-based private VM hosting; deployed 64TB Ceph cluster for private S3 storage; Submitted many upstream fixes to Ceph for S3 bugs; Network architecture work for multi-homed BGP & OSPF; VM migration from Ganeti to OpenStack research.
  • IsoHunt Web Technologies Inc. Lead Developer, Master Coder & Toolsmith. January 2006 - November 2013. Developed and deployed Internet software including next-gen targeted search engines and portals. Heavy focus in state-of-the-art text indexing and Web 2.0 systems complete stack scaling. Deployments spread over multiple datacentres with ownership extending from the hardware/kernel up to language optimizations and a special focus on MySQL. Managed codebases & teams working in PHP, Ruby on Rails, Perl, Python. Also provided specialized Gentoo system administration knowledge.
  • MogileFS (open-source/volunteer). 2007 - 2013. Developed and maintained the PostgreSQL backend for the MogileFS distributed file-system (deployed at scale for IsoHunt).
  • BC Libraries Cooperative/SITKA. Consultant System Administrator & Architect. March 2011 - December 2011. (consulting position developed into the full time position above).
  • Gentoo Council (open-source/volunteer). Elected member of the Gentoo Linux Council, 2006-2007 term. Responsible for major technical decisions in the distribution.
  • Global NetOptex, Inc. Consultant: Gentoo & MySQL. 2006 - 2010. Providing consulting services for Gentoo and MySQL deployments, with an emphasis on design, scaling and optimization.
  • Merkle; The Gallery Group; (end client Capital One Bank). Consultant: Gentoo, MySQL, iSCSI, OpenLDAP. February 2006 & March 2007. Providing consulting services for integration of Gentoo, iSCSI via Isilon, MySQL & OpenLDAP, with an emphasis on design, scaling and optimization (work previously under NDA).
  • Net-Conex Business Solutions Inc. Network Administrator. August 2005 - January 2006. Deployed and maintained large scale inter-building wireless backbone. Servicing 20+ buildings around Vancouver, using custom Linux routers (OSPF, iproute2, iptables). Deployment plan for VoIP utilizing wireless backbone.
  • phpMyAdmin (open-source/volunteer). Core Developer. May 2001 - May 2003, December 2003 - October 2005. Contributed to a major open source MySQL database management tool based in PHP. Implemented the database maintenance controls, table & engine type code, host-based authentication, SQL parser & syntax-highlighter, SQL validator and many other bugfixes & improvements. Represented phpMyAdmin at conferences: MySQL User Conference 2006-2011, OSCON 2006, LinuxTag Karlsruhe 2006.
  • Orbis-Terrarum Networks (not incorporated). Owner, System Administrator, Developer. Summer 2001 - 2005. Small business iteration of consulting services. Run Linux server hosting DNS, WWW, Mail & more. Developed Web-pages using PHP & MySQL. Custom software development & system administration services for a variety of open-source deployments.
  • Simon Fraser University Surrey, Research Assistant positions:
    • (Gordon Pritchard). System Administrator. January 2004 - August 2005. Setup & system administration of the research labs, comprising 30+ servers & 90+ workstations of mixed platforms (Gentoo Linux, Fedora Linux, SUSE Linux, Windows, MacOS, IRIX). Custom software development to aid system administration tasks.
    • (Kay Wiese). Linux Cluster development. June 2005 - August 2005. Complete development of a Gentoo Linux based High-Performance Cluster (128 nodes), including some development on GLI (Gentoo Linux Installer).
    • (Vive Kumar). Software Developer. April 2004 - August 2004. Design & development of a PKI & security solution for the ePortfolio project.
    • (John Nesbit). Zope & Linux consultant. September 2003 - April 2004. Consulted on Gentoo Linux server deployment and administration including multi-instance Zope installs.
    • (Gordon Pritchard). System Administrator. June 2003 - September 2003. Setup & maintenance of the new research labs, comprising 20+ servers & 70+ workstations.
    • (Vive Kumar, Marek Hatala). Software Developer. May 2003 - June 2003. Systems design for a data collection agent producing SCORM packages.
    • (Steve DiPaola). Software Developer. Summer 2003. Maintenance of medium-sized MFC application. Ported PHP-based application using PostGreSQL to use MySQL instead.
    • (Gordon Pritchard). Software Developer. September 2002 - April 2003. Programmed a real-time asset tracking system (RATS) in PHP & MySQL, running on Linux.
    • (Julie Tolmie). Lead programmer. May 2002 - December 2002. Lead programmer on open source mathematics visualization tool for SGI IRIX (MVS).
  • Technical University of British Columbia (Tom Calvert, Gordon Pritchard). Research Assistant (System Administrator). September 2001 - April 2002. Maintained research lab, running many operating systems, and help co-ordinate research projects.
  • AeroNett Technologies Ltd. Graphics Artist & Data Entry Specialist. March 2000. Graphics editing (Adobe Photoshop) and data entry work.
  • School District 41 (Burnaby), Computer Support Services division. Computer Technician. Q4 1999. Computer system repair, network maintenance.
  • Evergreen Communications. Graphics Artist. September 1999 - December 1999. Graphics and photo editing (Adobe Photoshop) for a Buddhist photo-book.
  • Orbit Computers. Owner. June 1997 - June 1998. Custom-assembled computers business, run while in high school in South Africa

Consulting

I offer consulting services in several fields, and am especially interested in intersections of the following fields.
  • S3 specification & protocol nuances
  • Ceph, esp. RGW deployment & scaling, for private & public clouds
  • Gentoo Linux integration & deployments
  • LDAP integration for Linux authentication
  • iSCSI clients on Linux

Published Papers & Conference Talks

  • Johnson, Robin H. & Maredia, Ali. RGW S3: Feature Progress, Limitations & Testing. Cephalocon 2019, Barcelona, Spain. May 20, 2019. (Slides, Video).
  • Johnson, Robin H. RGW S3: Features Vs Deep Compatibility. Cephalocon APAC 2018, Beijing, China. March 22, 2018. (Slides, Video)
  • Johnson, Robin H. Gentoo Foundation, background and status report. Gentoo Miniconf 2016, Prague. October 8th, 2016.
  • Johnson, Robin H. Gentoo Infra Roundtable. Gentoo Miniconf 2016, Prague. October 8th, 2016.
  • Rogan Hamby, Robin Johnson and Galen Charlton. There and Back Again, Again. Evergreen 2013, Vancouver. April 10-13, 2013.
  • Johnson, Robin H. Gentoo @ IsoHunt. Gentoo Miniconf 2012, Prague. October 20-21, 2012.
  • Johnson, Robin H. & Groffen, F., Council and Trustees: Managing Gentoo. Gentoo Miniconf 2012, Prague. October 20-21, 2012.
  • Johnson, Robin H. & Gentoo Infrastructure Team, State of Gentoo Infrastructure. Gentoo Miniconf 2012, Prague. October 20-21, 2012.
  • Johnson, Robin H. State of Gentoo BoF. Ottawa Linux Symposium. June 13-16, 2010.
  • Angenendt, R., Johnson, Robin H. Infrastructure round table. FOSDEM, Brussels. Feburary 6-7, 2010.
  • Johnson, Robin H. State of Gentoo BoF. Ottawa Linux Symposium. July 13-17, 2009.
  • Johnson, Robin H. State of Gentoo BoF. Ottawa Linux Symposium. July 23-26, 2008.
  • Lougheed P., Johnson R.H., Jordanov M., Bogyo B., Kumar V.S., Fee J., Securing electronic portfolios, International conference on ePortfolio, France, 2004.
  • Fee J., Lougheed P., Brokenshire D., Bogyo B., Jordanov M., Johnson R.H., Kumar V.S., ePortfolio: issues for research, International conference on ePortfolio, France, 2004.
  • Johnson, Robin H., Care & Feeding of phpMyAdmin and other veteran PHP applications. Vancouver PHP Conference. January 23rd, 2003.

Selected Past Projects

  • DigitalOcean: Alternate S3 gateway to replace RGW
  • DigitalOcean: S3 security, trust & safety tooling.
  • DigitalOcean: Capacity forecasting and planning for 50+ Ceph clusters
  • DreamHost: Ceph RGW S3 server rate-limiting via HAProxy-Lua.
  • Gentoo: Migration of main CVS tree to Git
  • BC Libraries Cooperative: Hyperconverged 64TB Ceph cluster w/ S3 API & Ganeti
  • IsoHunt: Web2.0 Distributed Text Indexing
  • Gentoo: Specification and development of the tree-signing proposal
    [GLEP57, GLEP58, GLEP59, GLEP60, GLEP61, GLEP74]
  • Gentoo: Migration of Bugzilla-2.x to 3.x
  • Porting Gentoo Linux & 2.4 kernel support for the MyCable XXS1500 (MIPS) to the 2.6 kernel.
  • Porting Gentoo Linux to SGI Visual Workstation 320 (VISWS@SourceForge, Gentoo-Wiki).
  • Capital One Bank: Gentoo LDAP integration with Isilon iSCSI
  • Distributed System Architecture For Random Walk Analysis (Paper for Distributed Systems class).
  • Comparision of Open Source Static-Analysis software tools (Paper for Computer Security class).

Open Source

I have contributed to numerous open source projects. I maintain a partial listing as a record of prior work for legal reasons.

My GitHub doesn't reflect much of my work, as it tends to be in other non-GitHub locations, like directly in Gentoo. But you can check out robbat2's GitHub Activity Feed anyway.


Knowledge

None of these are exhaustive lists, but are here for organizations that scan resumes by keyword.

I used to keep the following subsections more up to date, but then it became less relevant.
I became a toolsmith instead: Use the right tool for the job, learning as you go, and writing it when you need to.

Operating systems:
  • Linux (Gentoo, Ubuntu, Debian, RedHat, Fedora, Slackware, SuSE, Mandrake)
  • Windows 3.1/3.11WFW/9x/Me/NT/2K/XP
  • Mac OS X
  • FreeBSD 4/5/6/7
  • SGI IRIX 6.5
  • Novell Netware 3/4/5
  • DOS (3 and up)
A subset of my skills:
  • AJAX
  • Assembly (x86, PIC, MIPS)
  • Autoconf & Automake
  • Awk
  • BGP & OSPF (quagga, FRR, bird)
  • Ceph (RGW speciality)
  • Cfengine 2.x
  • Chef
  • C++
  • C
  • Clickhouse
  • Consul
  • Cumulus Linux
  • custom exporters
  • Golang
  • Go
  • Grafana
  • HP ProCurve switches
  • Icinga
  • iproute2
  • ipset
  • iptables
  • Java
  • JavaScript (& jQuery)
  • Katran
  • KVM (Ganeti, libvirt, QEMU)
  • M4
  • Minio
  • Nagios
  • node_exporter
  • NRPE
  • NTP
  • Perl
  • PHP
  • Prometheus
  • Puppet
  • Python
  • R
  • Ruby
  • S3
  • Sed
  • Shell (Bash, Zsh, Csh)
  • SQL (MySQL, PostGreSQL, Percona, MariaDB)
  • TCL/TK
  • XML (XPath, XSLT, XSchema, SOAP)
Certifications:

Education

Summary
  • Simon Fraser University, Fall 2002 - Winter 2005.
    Degree: Bachelor of Science (Information Technology, TechBC).
  • Technical University of British Columbia (TechBC), Fall 2001 - Winter 2002
    Information Technology

Languages

  • English (native)
  • Afrikaans (fluent)
  • Spanish (basic)
  • Latin (minimal reading)
  • Dutch, Italian, German, Polish (minimal survival)

References

Chad Heuchober
previously Senior Manager, Storage Engineering, DigitalOcean

Jonathan LaCour
CTO, Reliam / Mission Cloud
(previously Senior VP, Product & Technology, DreamHost)

Scott Leslie
Systems Manager, BC Libraries Cooperative

Gary Fung
CEO, isoHunt Web Technologies Inc.

Gordon Pritchard, P.Eng.
Research Lab Manager
Simon Fraser University @ Surrey

Last updated:2023/Feb/22