DevOps AWS Big Data (Java)
World’s Most Comprehensive Cross-Device Identity big data on Businesses, Consumers and Geo Audiences. When the number of records counts hundreds of billions and datasets are terabytes on dozens of instances, every optimization counts — either performance, or cost, or disk space, or stability & backups — everything is critical.
- perfect knowledge of Linux based OS and network protocols (iptables, ssh, HTTP, FTP, mail, LDAP, Samba, etc);
- perfect knowledge of AWS products (Amazon ES, EC2, VPC, VPN/ssh tunnel, ELB, S3, EMR, IAM, and Security) and infrastructure;
- experience with Docker;
- any monitoring tools (DataDog, Nagios, CloudWatch, Zabbix, Ganglia, etc.);
- ability to write bash scripts and automate infrastructure routine actions, writing from scratch and modifying config files;
- intermediate+ English (at least written);
- team player;
- a desire and ability of self-growth and self-education.
- knowledge of any programming language, preferably Java;
- experience with Elasticsearch, Kibana, and Logstash;
- experience with big data projects;
- knowledge of any Infrastructure as a Code tools (AWS Cloud Formation, Terraform, etc.);
- experience with log analysis tools (Splunk, Papertrail, etc.);
- good understanding of managed and not managed NoSQL databases and key-value storages;
- knowledge of Jenkins or any other CI/CD tools;
- pro-active approach of work in a mindset;
- process oriented;
- attention to details and accuracy;
- knowledge of Azure and MS Server, IIS, MS SQL Server;
- experience of the Hadoop related stack (HDFS, YARN, HBase, Spark);
- experience with Ansible;
- hands-on all AWS Re-Invent updates;
- ability to work overtime with the same performance.
- set up and configure EC2 Linux machines and related packages (Docker, Java, PHP, node.js, Databases, etc.);
- set up a virtual private network and secure access in AWS;
- set up monitoring and resolve incidents (out of space, out of memory, certificates expiration, etc.);
- set up and manage products AWS (Amazon ES, EC2, ELB, S3, EMR, etc.);
- write bash scripts and automate infrastructure routine actions;
- use (where possible) Infrastructure as a Code tools (AWS Cloud Formation, Terraform, etc.);
- set up and maintain CI/CD tools (Jenkins, etc).
- always think how to solve the task in the most cost-efficient way;
- propose actions to reduce infrastructure cost for existing accounts and implement them once it’s approved;
- follow all new AWS product announcements and suggest improvements to projects’ infrastructures;
- study AWS products and related technologies (monitoring tools, log analysis tools, databases, etc) to be able to help the team with configuration and improvements;
- monitor and keep tracking on abnormal cost spikes to answer the question of why it happened and what could be done to avoid that in future.
- on projects with a dedicated DevOps position (full-time or part-time) timely communicate with the customer in emails in English;
- in case of project infrastructure incident (going on or just resolved) participate in conf calls with customer biz or tech team;
- send planned maintenance email notifications;
- send emails on critical monitoring incidents’ fix status (sometimes with the investigated possible reason).