Experience

Apple, Inc.

Date of Employment: June 2016 - Present (6 years 9 months)

Type of Employment: FTE

Title: Site Reliability Engineer / Application Infrastructure Engineer

Facebook, Inc.

Date of Employment: February 2012 - March 2016 (4 years 2 months)

Type of Employment: Contractor (first 2 years) and FTE (last 2+ years)

Title: System Engineer

• 2+ years of experience working directly with deploying IPv6 in a large scale Linux environment.
• 4+ years of experience as a founding member of the IT Chef team.
  • Built IT Infra's multi-regional Chef infrastructure stack starting with the original implementation.
  • Lead engineer writing, testing, and implementing of the IT Chef code base.
  • Cookbook creation included basic system configurations to complex system automation workflows.
  • Orchestrated 600+ Chef code pushes involving cookbooks, roles, environments, and databags.
  • Authored Chef code to manage Chef itself to include managing its chef-clients on Chef nodes and auto-deploying Chef servers with Chef.
  • Authored an original password randomnizer in pure Ruby and integrated it with Chef seamlessly to securely randomize all root user passwords.
  • Authored a specialized piece of Chef automation code that dynamically dual-stacked RHEL and Ubuntu nodes with IPv6 network interfaces.
  • Authored all code to convert Chef cookbook based secrets to use encrypted data bags and migrated all Chef Linux nodes to support encrypted data bags.
  • Fully implemented a deployment of taste-tester, a Facebook specific Chef integration testing tool, on the IT Chef stack.
• Self taught the Ruby Programming language both as a declarative DSL within Chef and as a general OOP higher language.
• 4+ years of experience managing thousands of heterogeneous IT Linux systems (CentOS, RHEL, OEL, and Ubuntu) with Chef automation.
• 4+ years of hardening Chef managed Linux systems with things including: LDAP auth, DUO 2-factor auth, OpenSSH security tuning, and automated sudo policies.
• 4+ years of experience building and operating the multi-regional package management infrastructure for different Linux distributions.
  • Originally designed, built and implemented a multi-regional Linux driven HTTP heterogeneous package management service using Chef.
  • Authored an original package repository synchronization tool in pure Ruby that automatically mirrored public repos to keep them up-to-date.
  • Maintained and built all custom IT Infra packages that included RPM and DEB package artifacts.
  • Created a lightweight BASH package build script framework to assist in compiling source code based packages.
  • Mastered the usage of FPM, a ruby meta-packaging tool, that helped build custom code based packages.
  • Worked closely with Security teams to coordinate, push and mitigate packages affected by major security vulnerabilities.
  • Ensured all and any package spec files or custom build scripts were version controlled within git repositories.
• 2 years of experience building, re-designing, re-building, and operating multi-regional global Facebook IRC service.
  • Rebuilt and migrated the entire Facebook IRC Infra to the IT Infra stack, including different backend systems, load balancers, and networking.
  • Designed and implemented the re-designed IRC infrastructure entirely in and with Chef so that it was completely codified infrastructure.
  • Configured the IRC implementation to be a multi-faceted Internet facing service.
  • Iterated upon the IRC infrastructure by upgrading the whole stack to work with IPv6 making it the first Internet facing IPv6 IT service.
  • Coordinated and communicated IRC migrations, outages, and upgrades to the company.
  • Managed monitoring IRC service health, users, and traffic by failing regional traffic over during incidents and upgrades.
• 4+ years of experience in a fast paced, iterative development and operations environment.
• 3+ years of experience operating and managing load balancer traffic infrastructure for IT Infra services, focusing in Chef, Package Management and IRC services.
• 4+ years of experience provisioning virtualized Linux machines.
• 4+ years of experience with on-call rotations and on-demand repair of large scale service outages.
• 4+ years of experience with site/service event and incident response, incident root cause analysis, incident report creation, and incident review post mortem meetings.
• 4+ years of experience working in line with Technical Project Managers to push and deliver innovative new services.
• 4+ years of experience working with internal and external auditors to ensure compliance of system configurations for financial based infrastructure.
• 4+ years of experience rebuilding entire infrastructure stacks to migrate and upgrade them to different datacenters locations.
• 2+ years of experience actively participating in Disaster Recovery drills on core infrastructure services to test their resiliency and availability.
• Participated in complete Infrastructure Destruction and Security Breach simulations. Also, know as Red Team exercises.

Google, Inc.

Date of Employment: September 2010 - January 2012 (1 year 5 months)

Type of Employment: FTE

Title: Unix Systems Administrator

• Managed the engineering Ubuntu Linux development systems with BASH scripting, git code pushes, security package updates, and network monitoring.
• Converted the Slide Design Team to corporate Google hardware and software.
• Implemented a multiplatform SSHFS software deployment for Slide Teams to securely access remote file systems.
• Converted all Slide Team members to use SSH key based authentication when accessing a bastion hosts.
• Created a scripted account creation automation and management process for newly hired, exited, and existing Slide AU employees.
• Improved Slide AU account setup with multiple BASH scripts for LDAP auditing, developmental playground initialization and conversion, file encryption, and Slide AU specific OS configurations installations.
• Authored procedural documentation on Slide AU user account management, developer playground creation, SSH usage and ethics guide, new hires, terminations.
• Assisted the Slide AU Systems Operations team with a major datacenter colocation migration and setup.
• Supported multiple remote teams in China and Europe.

Slide, Inc.

Date of Employment: July 2009 - August 2010 (1 year 2 months)

Type of Employment: Contractor/FTE

Title: Systems Operations Admin and Facilities Assistant

• Worked on the Systems Operations team with an emphasis on Linux systems administration.
• Maintained, operated, and repaired corporate Ubuntu Linux servers and services.
• Deployed FOSWiki, FreeGhost/FOG PXE imaging system, and Apple Netboot System Image Server to improve desktop and laptop distribution.
• Administered the Mailman mailing lists, TWiki, and Kwok Information internal servers and services.
• Supported 100+ local employees and multiple overseas offices using mixed operating systems environments.
• Worked with Slide Facilities to help maintain, organize, and improve the three company office building floors.

Slide, Inc.

Date of Employment: July 2007 - September 2007 (3 months)

Type of Employment: Internship

Title: System Operations Intern

• Collaborated with Systems Operations team members to support all Slide employee desktop and laptop computers.
• Assisted in the physical hardware and software configuration maintenance of corporate Linux servers.
• Collaborated with network engineering to fix and setup new corporate network infrastructure.
• Implemented a new company wiki with TWiki to help improve employee communication, productivity, and sharing of project critical information.

Cal Poly Pomona University Library

Date of Employment: October 2005 - June 2007 (1 year 9 months)

Type of Employment: On-campus Student

Title: Student Systems Operations Admin

• Supported all library employees from six internal departments.
• Effectively maintained and ensured the functional and operational integrity of all computer labs consisting of 100+ computers of mixed operating systems.
• Completed tasks to include resolving support tickets in an efficient manner.
• Solved daily on-demand desktop software and user support problems for library employees.