When I start Python development on a new Apple, I immediately hit two problems:
I need a version of Python that’s not installed.
I need to install a bunch of packages from PyPI for ProjectA and a different bunch for ProjectB.
Virtualenv is not the answer! That’s the first tool you’ll hear about but it only partially solves one of these problems. You need more. There are a ton of tools and a ton of different ways to use them. Here’s how I do it on Apple’s Mac OS X.
If you’re asking questions like, “Why do you need multiple versions installed? Isn’t latest enough?” or “Why not just pip install all the packages for ProjectA and ProjectB?” then this article probably isn’t where you should start. Great answers to those questions have already been written. This is just a disambiguation page that shows you which tools to use for which problems and how to use them.
Installing Python Versions
I use pyenv, which is available in homebrew. It allows me to install arbitrary versions of Python and switch between them without replacing what’s included with the OS.
Note: You can use homebrew to install other versions of Python, but only a single version of Python 2 and a single version of Python 3 at a time. You can’t easily switch between two projects each frozen at 3.4 and 3.6 (for example). There’s also a limited list of versions available.
Install pyenv:
$ brew update
$ brew install pyenv
Ensure pyenv loads when you login by adding this to ~/.profile:
$ eval "$(pyenv init -)"
Activate pyenv now by either closing and re-opening Terminal or running:
$ source ~/.profile
List which versions are available and install one:
$ pyenv install --list
$ pyenv install 3.6.4
If the version you wanted was missing, update pyenv via homebrew:
$ brew update && brew upgrade pyenv
If you get weird errors about missing gcc or zlib, install the XCode Command Line Tools and try again:
$ xcode-select --install
I always set my global (aka default) version to the latest 3:
$ pyenv global 3.6.4
Update 2018-10-23: If I need several versions available, for example to run tests in tox:
$ pyenv global 3.6.4 3.7.0
Setting these makes versioned Python commands available:
$ python3.6 --version
$ python3.7 --version
Pyenv has lots of great features, like support for setting a different version whenever you’re in a specific directory. Check out its commands reference.
Installing PyPI Packages
In the old days, virtualenv was always the right solution. Today, it depends on the version of Python you’re using.
Python <= 3.3 (Including 2)
This is legacy Python, when environment management wasn’t native. In these ancient times, you needed a third party tool called virtualenv.
This installs the virtualenv Python package into the root environment for the legacy version of Python I need, then creates a virtual Python environment where I can install project-specific dependencies.
Python >= 3.3
In PEP 405 an environment manager called venv was added to core. It works pretty much like virtualenv.
Note: Virtualenv works with newer versions of Python, but it’s better to use a core library than to add a dependency. I only use the third party tool when I have to.
Last year, my buddy Kai had An OpeSec Event. His business depends on two dozen hosted services (a common situation today), and in the management of all that he’d forgotten to remove access for a former user. They deleted a bunch of stuff and he had to recover from backups. To prevent that from happening again, I shared a monthly checklist to run through for each service:
Can I cancel this service?
Should I delete users?
Should I change shared passwords?
Should I un-share anything?
Should I force-disconnect any devices?
Is the domain name about to expire?
Is the credit card about to expire?
Am I paying for more than I use?
Should I cancel auto-renewal?
Is the last backup bigger than the one before it?
Recently, while managing my own services, I realized I’d forgotten an item:
Are there any messages from the provider in my account?
Most services will email you updates, but I’ve learned not to depend on that. Last year there was an alert in the summary page of one of my investment accounts. It required action, but only appeared on that page. No email, no text message, no call. Once, a critical alert got filtered as spam and I barely caught it. Another time it turned out my email address had been updated by a customer service rep and they made a small typo, so I wasn’t getting alerts at all.
It’s a few minutes of work, but I log in to each provider account once a month and look for messages. This saves me from mistakes, but I also find value in seeing all my accounts personally on a regular schedule. It keeps me informed.
Today’s post is a little outside my usual DevOps geekery, but it’s been an influencer on my work and my career choices this year so I wanted to share it.
For the record, I have zero connections to 3M.
In my teens, I noticed that whenever I bought something with the 3M logo it was noticeably better than the other brands. I didn’t know what 3M was, but this pattern kept repeating and I started to always choose them. Years later, deep inside a career in technology, I was still choosing 3M. I started to ask myself how they did it. Why were all their products better than everyone else’s?
Balance? At work? And still better than everyone else? Bring it on.
The book approaches 3M through their innovations. They built hugely successful product lines in everything from sandpaper to projectors, and it turns out other companies have long looked to them as the top standard for the innovation that drives such diverse success. As I worked through the book, one thing really stuck with me: 3M’s definition of Stretch Goals.
I’ve seen a lot of managers ask their teams what can be accomplished in the next unit of time (sprint, quarter, etc.). Often, the team replies with a list that’s shorter than the manager would like. The manager then over-assigns the team by adding items as “stretch goals”. If the team works hard enough and accomplishes enough, they’ll have time to stretch themselves to meet these goals. The outcome I usually see is pressure for teams to work longer hours (with no extra pay) so they can deliver more product (at no extra cost to the company).
This book described 3M’s stretch goals very differently, which I’ll summarize in my own words because it’s characterized throughout the book and there’s no single quote that I think captures it. 3M sets these goals to stretch an aspect of the business that’s needed for it to remain a top competitor, and they’re deliberately ambitious. For example, one that 3M actually used: 30% of annual sales should come from products introduced in the last four years. Goals like these drive innovation because they’re too big to meet with the company’s current practices.
The key difference is that 3M isn’t trying to stretch the capacity of individuals. They’re not trying to increase Scrum points by pushing everyone to work late. They’re setting targets for the company that are impossible to meet unless the teams find new ways to work. They’re driving change by looking for things that can only be done with new approaches; things that can’t be done just by working longer hours. And after they set these goals, they send deeply committed managers out into the trenches to help their teams find and implement these changes. Most of the book is about what happens in those trenches. I highly recommend it.
There’s one other thing from the book I want to highlight: the process of innovation doesn’t simplify into management practices you can choose off a menu. There’s more magic to it than that. It takes skilled leaders and a delicate combination of freedom and pressure to build a company where the best engineers can do their best work, and trying to reduce that to a prescription doesn’t work. Here’s a quote from Dick Lidstad, one of the 3M leaders interviewed for the book, talking about staff from other companies who come to 3M looking to learn some of the innovation practices so they can implement them in their own teams:
They want to take away one or two things that will help them to innovate. … We say that maintaining a climate in which innovation flourishes may be the single biggest factor overall. As the conversation winds down, it becomes clear that what they want is something that is easily transferable. They want specific practices or policies, and get frustrated because they’d like to go away with a clear prescription.
I heard truth in that quote. Despite being a believer in the value of tools like Scrum, which are supposed to foster creativity and innovation, I’ve spent a lot of my career held back by the overhead of process that’s good in principle but applied with too little care to be effective. Ever spent an entire day in Scrum ceremonies? There’s more value in the experience of 3M’s teams overall than there is in any list of process.
This book was written in 2000, but not only has 3M stock continued to perform well, I found many parallels in the stories this author tells and my own experience in the modern tech world. It’s heavy with references and first-hand interviews, and I think it’s a valuable read for anyone in tech today.
data "external" "cars_count" {
program = ["python", "${path.module}/get_cool_data.py"]
query = {
thing_to_count = "cars"
}
}
output "cars_count" {
value = "${data.external.cars_count.result.cars}"
}
A slick, easy way to drop out of terraform and use Python to grab what you need (although it can get you in to trouble if you abuse it).
The Python script has to follow a protocol that defines formats, error handling, etc. It’s minimal but it’s fiddly, plus if you need more than one external data script it’s better to modularize than copy and paste, so I wrote a pip-installable decorator that implements the protocol for you. The source is also an example you can follow if you’d rather implement it yourself than add a dependency. Here’s how you use it:
from terraform_external_data import terraform_external_data
@terraform_external_data
def get_cool_data(query):
return {query['thing_to_count']: '3'}
if __name__ == '__main__':
get_cool_data()
It’s available on PyPI, just pip install terraform_external_data.
While studying for an AWS certification, I rediscovered a fiddly networking detail: although ICMP’s ping is stateless, EC2 security groups will pass return ping traffic even when only one direction is defined in their rules. I wanted to see this in action, so I built a lab.
If you just asked, “Wat❓”, keep reading. Skip to the next section if you just want the code.
Background
Network hosts using stateful protocols (like TCP) distinguish between packets that are part of an established connection and packets that are new. For example, when I SSH (which runs on TCP) from A to B:
A asks B to start a new connection.
B agrees.
A and B exchange bunches of packets that are part of the connection they agreed to.
A and B agree to close the connection.
There’s a difference between a new packet and a packet that’s part of an ongoing connection. That means the connection, and its packets, have state (e.g. new vs established). Stateful firewalls (vs stateless) are aware of this:
A ask B to start a new connection.
Firewalls in between allow these packets if there is an explicit rule allowing traffic from A to B.
A and B exchange bunches of packets.
Firewalls in between allow the packets from A to B because of the explicit rule above. However, they allow the return traffic from B to A even if there is no explicit rule to allow it. Since B agreed to the connection the firewall assumes that packets in that connection should be allowed.
In EC2, this is why you only need an outgoing rule on A’s Security Group (SG) and an incoming rule on B’s Security Group to SSH from A to B. EC2 SGs are stateful, and allow the return traffic implicitly.
Ok, here’s the gnarly bit. ICMP (the protocol behind ping) is stateless. Hosts don’t have a negotiation phase where the agree to establish a connection. They just send packets and hope. So, doesn’t that mean I need to write explicit firewall rules in the SGs to allow the return traffic? If the firewall can’t see the state of the connection, it won’t be able to implicitly figure out to allow that traffic, right?
Nope, they infer state based on timeouts and packet types. ICMP pings are ECHO requests answered by ECHO replies. If the SG has seen a request within the timeout, it makes the educated guess that replies are essentially part of “established” connections and allows them. This is what I wanted to see in action.
The Lab
I setup a VPC with two hosts, A (10.0.1.70) and B (10.0.2.35). They’re in different subnets but the ACLs allow all traffic so they don’t influence the test. Here are the SG rules for A (10.0.0.0/16 covers the entire VPC):
And the rules for B:
A allows outgoing ICMP to B, and B allows incoming ICMP from A. The return traffic is not allowed by any rules.
The Test Script
I didn’t find a way to send just replies without requests in Linux, so I bodged together a Python script:
"""
This is a stripped-down version of ping that allows you to send a reply without responding a request. This was needed
to test the details of how Amazon EC2 security groups handle state with ICMP traffic. You shouldn't use this for normal
pings.
The ping implementation was based on Samuel Stauffer's python-ping: https://github.com/samuel/python-ping (which only
works with Python 2).
This must be run as root.
You must tell the Linux kernel to ignore ICMP before you run this or it'll eat some of the traffic:
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all
"""
import argparse, socket, struct, time
def get_arguments():
parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
parser.add_argument('--send-request', metavar='IP_ADDRESS', type=str, help='IP address to send ECHO request.')
parser.add_argument('--receive', action='store_true', help='Wait for a reply.')
parser.add_argument('--send-reply', metavar='IP_ADDRESS', type=str, help='IP address to send ECHO reply.')
return parser.parse_args()
def receive(my_socket):
while True:
recPacket, addr = my_socket.recvfrom(1024)
icmpHeader = recPacket[20:28]
icmp_type, code, checksum, packetID, sequence = struct.unpack("bbHHh", icmpHeader)
print('Received type {}.'.format(icmp_type))
def ping(my_socket, dest_addr, icmp_type):
dest_addr = socket.gethostbyname(dest_addr)
bytesInDouble = struct.calcsize("d")
data = (192 - bytesInDouble) * "Q"
data = struct.pack("d", time.time()) + data
dummy_checksum = 1 & 0xffff
dummy_id = 1 & 0xFFFF
# Header is type (8), code (8), checksum (16), id (16), sequence (16)
header = struct.pack("bbHHh", icmp_type, 0, socket.htons(dummy_checksum), dummy_id, 1)
packet = header + data
my_socket.sendto(packet, (dest_addr, 1))
if __name__ == '__main__':
args = get_arguments()
icmp = socket.getprotobyname("icmp")
my_socket = socket.socket(socket.AF_INET, socket.SOCK_RAW, icmp)
if args.send_request:
ping(my_socket, args.send_request, icmp_type=8) # Type 8 is ECHO request.
if args.receive:
receive(my_socket)
if args.send_reply:
ping(my_socket, args.send_reply, icmp_type=0) # Type 0 is ECHO reply.
You can skip reading the code, the important thing is that we can individually choose to listen for packets, send ECHO requests, or send ECHO replies:
python ping.py --help
usage: ping.py [-h] [--send-request IP_ADDRESS] [--receive]
[--send-reply IP_ADDRESS]
optional arguments:
-h, --help show this help message and exit
--send-request IP_ADDRESS
IP address to send ECHO request. (default: None)
--receive Wait for a reply. (default: False)
--send-reply IP_ADDRESS
IP address to send ECHO reply. (default: None)<span data-mce-type="bookmark" id="mce_SELREST_start" data-mce-style="overflow:hidden;line-height:0" style="overflow:hidden;line-height:0" ></span>
The Experiment
SSH to each host and tell Linux to ignore ICMP traffic so I can use the script to capture it (see docstring in the script above):
sudo su -
echo 1 > /proc/sys/net/ipv4/icmp_echo_ignore_all
exit
Normal Ping
I send a request from A to B and expect the reply from B to A to be allowed. Here’s what happened:
Remember A is 10.0.1.70 and B is 10.0.2.35.
Ok, nothing surprising. I sent a request from A to B and started listening on A, a little later I send a reply from B to A and it was allowed. You can do this test with the normal Linux ping command (but not until you tell the kernel to stop ignoring ICMP traffic). This test just validates that my bodged Python actually works.
Reply Only
First we wait a bit. The previous test sent a request from A to B, which started a timer in the SG. Until that timer expires, reply traffic will be allowed. We need to wait for that expiration before this next test is valid.
Boom! I start listening on A, without sending a request. On B I send a reply to A but it never arrives. The Security Group didn’t allow it. This demonstrates that EC2 Security Groups are inferring the state of ICMP pings by reading their type.
Other Tests
I also tried a couple other things that I’ll leave to you to reproduce in your own lab if you want to see them.
Start out like the normal test. Send a request from A to B and start listening on A. Then send several replies from B to A. They’re all allowed. This shows that the SG isn’t counting to ensure it only allows one reply for each request; if it has seen just one request within the timeout it allows replies even if there are multiple.
Edit the script above to set the hardcoded ID to be different on A than it is on B. Then nothing works at all. I’m not actually sure what causes this. Could be that the SG is looking at more than just the type, but it could also be in the kernel or the network drivers or somewhere else I haven’t thought of. If you figure it out, message me!
Conclusion
I had free time over the holidays this year! Realistically, understanding this demo isn’t a priority for doing good work on AWS. I just enjoy unwrapping black boxes to see how the parts move.
I’ll be at AWS re:Invent 2017! I’m there Monday through Wednesday, and I’ll be running Tuesday morning to support Girls Who Code and the American Heart Association. Get in touch if you’ll be at re:Invent or in the run so we can find each other!
🏃 Update: I finished the race at an 8:14 min/mile pace, which beats last year’s pace! But, this year’s run was shorter. Still, there was a huge turnout and it was a ton of fun.
Recently I’ve been working on the AWS Certification exams, and I’ve found they require much deeper understanding of networking on the platform than I had. For example, ICMP is a stateless protocol, so to ping between two servers do you need ingress and egress rules on both Security Groups? I knew from past experience with iptables that the answer varies by setup, but I didn’t know how it worked in EC2.
For me, gnarly networking is easiest to learn hands-on. Docs get me part of the way but I really need to engineer it myself before I’ll remember it. To prep for certification I ended up building a sandbox environment in my AWS account where I could play around. It took some doing; many AWS patterns come pre-baked with Security Groups, ACLs, etc. that make everything work, but I wanted everything turned off so I could verify what was really needed for different traffic flows. If I delete the egress rule on one side of a connection, does traffic still flow? Hard to validate if there are broad, generic rules in place. Easy to validate if only exactly what’s needed is present.
Since it was tricky, I published the automation for the sandbox I’ve been using. If you want to do your own deep dive of networking in AWS, hopefully this will help you out.
Production is hard. Even a simple script that looks up queue state and sends it to an API gets complex in prod. Without tests, the divide by zero case you missed will mask queue overloads. Someone won’t see that required argument you didn’t enforce and break everything when they accidentally publish a null value. You’ll forget to timestamp one of your output lines and then when the queue goes down you won’t be able to correlate queue status to network events.
Python can help! Out of box it can give you essential but often-skipped features, like these:
Automated tests for multiple platforms.
A --simulate option.
Command line sanity like a --help option and enforcement of required arguments.
Informative log output.
An easy way to build and package.
An easy way to install a build without a git clone.
A command that you can just run like any other command. No weird shell setup or invocation required.
It can be a little tricky, though, if you haven’t already done it. So I wrote a project that demonstrates it for you. It includes an example of a script that isn’t ready for prod.
Hopefully this will save you from some of the many totally avoidable, horrible problems that bad scripts have caused in my prods.
Update 29 September 2018: Since the release of AWS Step Functions, this pattern is out of date. Orchestrating deployment with a slim lambda function is still a solid pattern, but Step Functions could make the implementation simpler and could provide more features (like graphical display of deployment status). You can even see some of the ways this might work in AWS’s DevOps blog. Those implementation changes would drive some re-architecture, too. Because those are major changes, for now Pear is an interesting exploration of a pattern but isn’t ready for use in new deployments.
A while back I wanted to put a voice interface in front of my deployment automation. I think passwords on voice interfaces are annoying and aren’t secure, so I wanted an unauthenticated system. I didn’t want to lay awake at night worried about all the things people could break with it, so I set out to engineer a deployment infrastructure that I could put behind voice control and still feel was secure and reliable.
After a lot of learning (the journey towards this is where the Life Coach and the Better Alexa Quick Start came from) and several revisions, I had built my new infrastructure. For reasons I can’t remember, I called it Pear.
Pear is designed to make it easy to add slick features like voice interfaces while giving you enough control to stay secure and enough stability to operate in production. It’s meant for infrastructures too complex to fit in tools like Heroku or Elastic Beanstalk, for situations where you need to be able to turn all the knobs. I think it achieves those goals, so I decided to publish it. Check out the repo for an example implementation and more complete documentation of its features, but to give you a taste here’s a diagram of the basic architecture:
You’ve seen the Life Coach. That was my second Alexa project. The one I used to learn the platform.
I began with Amazon’s quick start tutorial, but I didn’t like the alexa-skills-kit-color-expert-python example code. It feels like a spaghetti-nest and is more complicated than it needs to be for a quick start (it’s 200 lines, the Life Coach is 40).
Hoping to make the intro process smoother for you, I’ve turned the Life Coach into my own quick start for Alexa. It gives you an example that’s simple but still has the tools you need to do your own development. Essential things like error log capturing and scripts to build and publish code updates that aren’t included in the Amazon guide.
Especially without Ops or AWS background, setting up AWS well enough that you don’t want to pull out your hair can be harder than writing your first skill. I’ve included automation to handle the infrastructure. You can skip that and just use the Life Coach code if you’d rather.
Hopefully this makes your Alexa journey easier. Check it out!
If you want an intro to how Alexa works before you dive in to code, check out the slides from my lighting talk on this project at the May 2017 meeting of the San Diego Python group.
