Welcome! This is the fourth edition of CloudProfs, sent to subscribers on August 20. See the email in-browser here.
If you enjoyed this newsletter, why not sign up here? Or if you have any feedback, email the editor here.
News
Google Cloud certifications offer the most likely route to a top-paying salary, according to a new study. The figures come from Global Knowledge and are based on responses from more than 3,700 US-based workers. Being a Google-certified professional data engineer offers an average salary of $171,749 (US), compared with Google-certified professional cloud architect at $169,029. AWS Certified Solutions Architect Associate comes in at $159,033.
The report also explored the average profile of a top earner. The average Google-certified professional data engineer has four certs on average, is most likely a Cloud Architect, and is also most likely to hold an AWS certification. Get the full data here.
Linkerd, a service mesh, is the latest project to graduate through the Cloud Native Computing Foundation (CNCF). The move indicates that Linkerd has hit a set level of maturity both in product and customer acquisition, with customers including Nordstrom, Expedia and Microsoft. William Morgan, CEO of Linkerd creator Buoyant, said the company was ‘humbled and honored’ and promised to be ‘unwavering in [the] mission to keep Linkerd simple, lightweight, and fast.’ A strong part of this ethos is down to Linkerd being built in Rust.
A couple of steps back, the CNCF confirmed this week that KEDA has been accepted as an incubating project. KEDA, which stands for Kubernetes Event-Driven Autoscaling, is a ‘single-purpose event-driven autoscaler for Kubernetes that can be easily added to Kubernetes clusters to scale applications.’
Amazon Web Services has launched the AWS Innovation Sandbox to help users quickly and safely experiment on the platform. The service ‘provisions isolated, self-contained environments to help developers, security professionals, and infrastructure teams to securely evaluate, explore, and build POCs using AWS services and third-party applications that run on AWS.’ The AWS solutions implementations page offers technical reference implementations vetted by AWS architects, offering detailed architecture and instructions for deployment.
Jobs
If you are actively looking to move elsewhere, looking for the sorts of skills required to take the next step in your career, or just interested in other organisations’ tech stacks, take a look at these interesting job postings. Please note all of these openings are US-based but are advertised as remote.
Starbucks is looking for a Principal Cloud Data Architect. The ideal candidate needs 10+ years of overall IT experience; experience with cloud performance and cost optimization; and a ‘demonstrated record of technical expertise and leadership.’ The candidate also needs expertise in two or more of the following services: SQL (Azure SQL, AWS RDS, PostgreSQL, MySQL/MariaDB), NoSQL (Cosmos DB, DynamoDB, MongoDB), Advanced Analytic Platforms (Azure Synapse, AWS Redshift, Databricks, Snowflake). The candidate must be fluent in Python.
Slack is hiring a Senior Staff Cloud Engineer – Cloud Infrastructure. The role entails leading engineering efforts on Slack’s underlying cloud infrastructure and beyond, including internal PaaS compute options (Kubernetes, Chef and VM, Serverless) as well as Amazon Web Services. The ideal candidate, with at least 10 years’ experience in software engineering, must ‘consider themselves an expert in a number of AWS services’; a proven track record in building tooling, automation in one or multiple languages (e.g. Go, Python), and have familiarity with tools such as Chef, Terraform, Ansible, or CloudFormation.
The Public Library of Science (PLOS) has a vacancy for a Cloud Software Engineer. The ideal candidate will be a ‘software generalist who builds scalable and highly available applications using best practices and tools that leverage fully managed services on Google Cloud Platform.’ The candidate needs to be fluent in either Python or Java, proficient in JavaScript and Bash, and have at least three years’ professional software engineering experience.
On the vendor side?
Amazon Web Services is recruiting for a Solutions Architect in its gaming team.
NetApp is looking for a Cloud Solutions Architect (remote but has to be within US-West time zone)
Adobe, Oracle and VMware are actively recruiting for architecture roles right now.
Tutorial and tests: QuestDB vs TimescaleDB
Kovid Rathee writes on a test between QuestDB and TimescaleDB using the Time Series Benchmark Suite (TSBS). TSBS is a collection of Go programs used to generate datasets and then benchmark read and write performance of databases. Currently, TSBS supports DevOps and IoT use cases.
Docker Hub was used to run the test, which tested both databases in four phases: generating DevOps data for one day, loading the generated data, generating queries to run on the loaded data, and executing the generated queries. A full output of the code used to run the tests can be seen here.
The results for loading data:
TimescaleDB
./tsbs_load load timescaledb –config=./config.yaml
…
Summary:
loaded 174528000 metrics in 59.376sec with 8 workers (mean rate 2939386.10 metrics/sec)
loaded 15552000 rows in 59.376sec with 8 workers (mean rate 261925.49 rows/sec)
QuestDB
./tsbs_load_questdb –file /tmp/questdb-data –workers 8
…
Summary:
loaded 174528000 metrics in 18.223sec with 8 workers (mean rate 9577373.06 metrics/sec)
loaded 15552000 rows in 18.223sec with 8 workers (mean rate 853429.28 rows/sec
QuestDB was approx. 3.2 times faster than TimescaleDB.
The results for executing queries:
TimescaleDB
Run complete after 1000 queries with 8 workers (Overall query rate 25.78 queries/sec):
TimescaleDB CPU over threshold, all hosts:
min: 222.49ms, med: 274.94ms, mean: 308.60ms, max: 580.13ms, stddev: 73.70ms, sum: 308.6sec, count: 1000
all queries :
min: 222.49ms, med: 274.94ms, mean: 308.60ms, max: 580.13ms, stddev: 73.70ms, sum: 308.6sec, count: 1000
wall clock time: 38.827242sec
QuestDB
Run complete after 1000 queries with 8 workers (Overall query rate 70.18 queries/sec):
QuestDB CPU over threshold, all hosts:
min: 92.71ms, med: 109.10ms, mean: 113.32ms, max: 382.57ms, stddev: 19.34ms, sum: 113.3sec, count: 1000
all queries :
min: 92.71ms, med: 109.10ms, mean: 113.32ms, max: 382.57ms, stddev: 19.34ms, sum: 113.3sec, count: 1000
wall clock time: 14.275811sec
QuestDB was approx. 2.7 times faster than TimescaleDB.
BONUS LINK: Read more from QuestDB: Comparing InfluxDB, TimescaleDB, and QuestDB timeseries databases
Understanding… sysbench
Continuing the theme of database benchmark testing, this article from Anthony Monthe, cloud benchmarker at Cloud Mercato looks at sysbench, a scriptable multi-threaded benchmark tool based on LuaJIT. It can also be used to evaluate cloud server performance. Used with kind permission of the author.
Among all benchmark tools available online, sysbench is one of my favorite. If I have to give two words to it, it would be simplicity and reliability. At Cloud Mercato we mainly focus on three of its parts:
- CPU: Simple CPU benchmark
- Memory: Memory access benchmark
- OLTP: Collection of OLTP-like database benchmarks
On top of that, sysbench can be considered as an agnostic benchmark tool, like wrk, it integrates a LuaJIT interpreter allowing to plug the task of your choice and thus to obtain rate, maximum, percentile and more. But today, let’s just focus on the CPU benchmark aka prime number search.
Run on command line
sysbench –threads=$cpu_number –time=$time cpu –cpu-max-prime=$max_prime run
The output of this command is not terrifying:
sysbench 1.1.0 (using bundled LuaJIT 2.1.0-beta3)
Running the test with following options:
Number of threads: 8
Initializing random number generator from current time
Prime numbers limit: 64000
Initializing worker threads…
Threads started!
CPU speed:
events per second: 707.94
Throughput:
events/s (eps): 707.9361
time elapsed: 30.0112s
total number of events: 21246
Latency (ms):
min: 11.25
avg: 11.30
max: 19.08
95th percentile: 11.24
sum: 240041.14
Threads fairness:
events (avg/stddev): 2655.7500/1.79
execution time (avg/stddev): 30.0051/0.00
The most interesting value here is the “events per second” or prime number found per second. But aside of that the output is pretty human readable and very easy to parse. It contains the base stats required for a benchmark in term of context and mathematical aggregations.
But what did I do?
That being said, as sysbench is open-source and available on GitHub, with a beginner level in C, it’s not hard to understand it. The code sample below represents the core of what is timed during the testing:
int cpu_execute_event(sb_event_t *r, int thread_id)
{
unsigned long long c;
unsigned long long l;
double t;
unsigned long long n=0;
(void)thread_id; /* unused */
(void)r; /* unused */
/* So far we’re using very simple test prime number tests in 64bit */
for(c=3; c < max_prime; c++)
{
t = sqrt((double)c);
for(l = 2; l <= t; l++) if (c % l == 0) break; if (l > t )
n++;
}
return 0;
}
Source: GitHub akopytov/sysbench
Basically if I translate the function in human words it would be: loop over numbers and check if they are divisible only by themselves.
And yes, the deepness of this benchmark is just these 15 lines, simple loops and some arithmetic. For me, this is clearly the strength of this tool: Again simplicity and reliability. sysbench cpu crossed the ages and as it makes more than 15 years that it didn’t changed, it allow you to compare old chips like Intel Sandy Bridge versus the latest ARM.
It is a well coded prime-number test, but what I see is that there isn’t any complex things developers often setup to achieve their goals such as thread cooperation, encryption or advanced mathematics. It just does a unique task representing a simple “How fast are my CPUs?”
The RAM involvement is almost null, processors’ features generally don’t improve performance for these kind of tasks and for a Cloud Benchmarker like me, this is essential to erase as most bias as possible. Where I can get performance data from complex workloads and have an idea of the system capacity, sysbench captures a raw strength that I correlate later with more complex results.
How do I use it?
The only one parameter entirely specific to sysbench cpu is max-prime, it defines the highest prime number during the test. Higher this value is, higher will be the time to find all the prime numbers.
Our methodology considers an upper limit of 64000 and a scaling up of thread number, from 1 to 2x the number CPU present on the machine. It produces data like in the chart above where we can see that the OTC’s s3.8xlarge.1 has the ability to scale at 100% until 32 threads which is its physical limit.
Take a look at a benchmark example here.
Complexities in 5G and edge computing architectures
Sagar Nangare, of Coredge.io, takes a theoretical approach on the synergies and struggles for 5G and edge computing architectures. Used with kind permission from the author.
Upcoming 5G networks are going to make significant changes in the lives of many people. 5G networks will provide a better digital experience to subscribers for many lucrative use cases. It will be a faster network than ever before. It is characterized due to its software-based core that improves delivery of services, massive network capacity, increases availability, enables end-to-end automation, reduces latency, and provides high bandwidth. Such features further enable several use cases that majorly impact the digital experience of network consumers. For example, 5G opens the possibility to support use cases like connected vehicles, remote surgery, industrial IoT, gaming and so on.
Obstacles in 5G
The impact of 5G will be experienced by individuals and digital infrastructures in various business sectors like transportation, industrial operations, healthcare, and retail. 5G will be delivered mainly by telecom operators, while it can also be deployed privately for enterprises. Though the 5G benefits are common across all the use cases, the delivery model will be different.
But 5G will bring complexity in core infrastructure that replaces many network functions with software (virtualized or containerized) based functions that will be hosted on commercial off-the-shelf hardware units spanned across telecom networks. This is due to reasons like:
- Involvement of different vendors for software and hardware resources
- Management of an increased number of devices and wider scale of network
- Variety of applications from the cloud to edge
- Deployment and management of applications that containerized, VM based and implemented on top of bare metal in core cloud and edge infrastructures
- Security issues due to more devices and distributed cloud infrastructure
Considering these complexities involved, telecom operators are looking to achieve the high velocity service delivery, guaranteed SLAs along with the lowest latency for enabling use cases like video conferencing, remote surgery, autonomous cars, maintaining the observability to keep infrastructure updated and secure, and keeping the cost low for operations and resource investments.
Key resolutions for 5G concerns
In a roadmap to realize the features of 5G, telecom operators are looking to solve challenges as we have discussed. Traditionally, we look at telecom networks or enterprise WAN infrastructure consisting of legacy network components, RAN resources, radio network resources, and core network components from a limited set of vendors. But in the case of 5G edge-based networks, telecom operators are looking to completely have a greenfield network that leverages cloud-native stacks (containers, Kubernetes, microservices, service meshes) along with agile delivery architecture with DevOps and CI/CD methodologies.
Such a newly formed complex architecture that includes a diverse set of vendors and mesh of services requires a unified platform that can:
- Manage legacy virtualized network functions along with modern containerized
- Help orchestrate 5G applications to orchestrate and automate end-to-end using a single pane of glass
- Enables management of distributed cloud environments that can include edge and public cloud instances
- Provides comprehensive monitoring and security to applications and network workloads
- Unifies OSS integration for all the elements of 5G and edge architecture
BONUS: The long reads
Take some time out and get stuck in to these rich cloud articles from across the web:
Aug 19: AWS vs Azure vs GCP cloud comparison: Databases (A Cloud Guru)
Aug 18: Compare AWS Lambda vs Azure Functions vs Google Cloud Functions (SearchCloudComputing)
Aug 18: A deep dive into the core concepts of ApsaraDB for MongoDB (Alibaba Cloud)