Downgrade RDS from m3 classic to t2 small VPC

The AWS console does not directly allow you to downgrade RDS from m3 to t2 sizing.

Here’s how to resize a db.m3.medium RDS classic instance to a db.t2.small VPC instance in a few AWS console steps.


This article assumes the following:
* Your RDS instance is still running on the older “EC2-Classic” platform
* You already have an EC2 VPC set up (your EC2 instances are running in a VPC).
Read More

Resize m3 to t2 EC2 Instances on AWS

If you run a server on AWS, you may be unpleasantly surprised to discover you don’t have the option to automatically resize between all instance types. I decided to resize m3 to t2 instance types (m3.large => t2.small) when my reserved instance lease expired & my needs had changed from those of running a commercial web business to running a small blog.

Turns out, AWS has made numerous network and architecture changes since my machine image was created in 2014. What should have been a small change turned into quite a bit of work. I’ve summarized the steps below should you encounter the same issue.

EC2 instances built today (called EC2-VPC) do not work interchangeably with older generation (now called EC2-Classic) instances. Here are some key differences I encountered: Read More

AWS Command Line Interface (aws cli) Tips

Today I started playing with the new Amazon Web Services command line interface tools to issue aws commands from my console and scripts.

Installation was straightforward, but I realized right away I needed to set the target region for my commands, so I decided to use the custom config file approach, setting an env variable AWS_CONFIG_FILE to point to my config file path.

The aws cli tools are not very well documented yet, and there are multiple obsolete versions of the docs floating around as well, so here are a few quick corrections.

Regarding your AWS config file:

1. You must explicitly prefix named config sections with “profile”, e.g. [profile oregon], not [oregon].

If you do not, an otherwise valid config file does not work, yielding this error.

A client error (InvalidLocationConstraint) occurred: The specified location-constraint is not valid

Here’s a valid config file:

[profile oregon]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
#oregon
region = us-west-2

[profile norcal]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
#n. california
region = us-west-1

2. Inline comments are not supported in the config file, only full-line comments.

If you do use an inline comment in your config (as one of their examples does), you may see the error I saw:

A client error (InvalidLocationConstraint) occurred: The specified location-constraint is not valid

So this inline region comment is invalid:

[default]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = us-west-2 #oregon
 

3. There is no fallback to env variables if you skip variables in the config.

Even though I set AWS_ACCESS_KEY and AWS_SECRET_KEY into my environment, I get error

Unable to locate credentials

So here’s another bad file.

[default]
region = us-west-2 

#missing aws_access_key_id 
#missing aws_secret_access_key

Setting up SSL on Apache (with Ubuntu12 + AWS)

Today I added SSL to my Apache webserver, running on Ubuntu 12, on an AWS instance. This was the first time I’d ever worked with SSL or certificates and it was fairly straightforward though it seemed daunting at first. Ran into a few problems that the Internet didn’t solve for me, so I thought I’d share.

My sequence end to end:

1. When I bought my domain name through Namecheap, it came with an SSL certificate, which I had never activated. Namecheap apparently subcontracts SSL services to a company called Comodo.  You’ll presumably need to purchase a trusted certificate from an authority like Comodo or DigiCert if you don’t already have one for your production site.

2. In preparation for using SSL, I added HTTPS (port 443) to my EC2 Security Group to allow traffic through the firewall. This is in the AWS Management Console ( EC2 -> Security Groups in left nav -> select the group in use by your server instance, click Inbound tab, HTTPS is listed in the dropdown of pre-configured rules you can add).  Here’s what it looks like after you add it:

image

3. I followed the instructions here to generate a private key (myprivatekey.key) and csr file (myserver.csr), saving them to a special directory for safekeeping. Basically this consisted of running:

openssl req -nodes -newkey rsa:2048 -keyout myprivatekey.key -out myserver.csr

Note that I’m using Apache 2 with mod_ssl. Instructions for other OS/webserver configurations are here . 

4. I submitted my generated csr to Namecheap through their web form, clicked on an approval email they sent, then received my certificate files by email from Comodo. They sent me a zip containing 2 files:

  • www_myserver_com.crt
  • www_myserver_com.ca-bundle

5. Next I roughly followed the instructions here to setup the SSL files in production. For other OS/webserver configs, you can look here. For my setup, this consisted of

  • a. uploading myprivatekey.key file and the zip with the certificate files to AWS, then unzipping the certificate files
  • b. copying the private key under /etc/ssl/private
  • c. copying the 2 unzipped certificate files under /etc/ssl/certs

6.  Rather than mucking with Apache’s default config files, I typically load my own Apache .conf file that lives in: /etc/apache2/conf.d/mydomain.conf . To enable SSL, I edited mydomain.conf file, adding the block below.

#remove the space after the < brackets in the Virtual Host open/close tags. 
#   Tumblr forces me to add it.
<VirtualHost *:443>
 SSLEngine on
 ServerName myserver.com
 SSLCertificateKeyFile /etc/ssl/private/mysslprivatekey.key
 SSLCertificateFile /etc/ssl/certs/www_myserver_com.crt
 SSLCertificateChainFile /etc/ssl/certs/www_myserver_com.ca-bundle
</VirtualHost>

I already had entries for port 80, so I just had to add port 443. The Document Root and LogFile locations were inherited from elsewhere in my config, which was fine for my purposes.

7. Enabled ssl for Apache by symlinking the available module under the enabled modules directory, then restarted apache:

 $ pushd /etc/apache2/mods-enabled/
 $ sudo ln -s ../mods-available/ssl.conf ssl.conf
 $ sudo ln -s ../mods-available/ssl.load ssl.load
 $ sudo /usr/sbin/apachectl restart

8. restarted apache and tailed my access.log and error.log to check for problems.

tail -n30 /var/log/apache2/access.log
tail -n30 /var/log/apache2/error.log

Note that I had originally included the following lines in my .conf file (without the # signs to comment them out), but they caused problems.

#NameVirtualHost *:80
#NameVirtualHost *:443
#Listen *:80
#Listen *:443

I removed these lines because they broke Apache restart and yielded these errors:

[Wed Jun 26 23:04:57 2013] [warn] NameVirtualHost *:443 has no VirtualHosts
[Wed Jun 26 23:04:57 2013] [warn] NameVirtualHost *:80 has no VirtualHosts
(98)Address already in use: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down

Basically, because I left Apache’s default configuration in place and was using a supplementary conf file, my lines were duplicates of lines in Apache’s /etc/apache2/ports.conf file.  You’ll need the Listen port lines somewhere in your Apache configuration to get things working properly, but not loaded twice if you want everything to start!

I still see this additional warning in my error.log at startup, but safely disregard it since it does not impact functionality. I leave my instance hostname as AWS has configured it.

RSA server certificate CommonName (CN) `www.myserver.com' does NOT match server name!?

9. To verify everything was working properly with my SSL certificate, I first ran a check of my website’s certificate configuration here: http://www.digicert.com/help/

10. I then doublechecked that both of these urls worked for my server, and that my apache access.log showed requests with port 80 and port 443 in use respectively.

http://www.myservernamehere.com

https://www.myservernamehere.com

Next up: getting an SSL certificate to work in my dev environment, not nearly so straightforward it turns out.

This week in Startup Engineering: Decoupling Django DB & Web logic

In theory it sounded very straightforward, a simple refactoring.  

My goal: Separate the django database logic from the web/UI/business logic code. Out of the box, django worked like a charm, an all-in-one stack that ran very efficiently for a web/db prototype website on Amazon Web Services.

But in order to support future scalability, I needed to decouple these components, so they could live on the same or different servers transparently, and communicate completely through service APIs, a la the infamous Steve Yegge rant touting Jeff Bezos’s all-services-all-the-time mandate.

Things started simply enough, reviewing the existing views and models, figuring out what type of generic APIs I would need for a decoupled world.  Then it hit me. To separate user data from the web server meant an entirely new level of authentication and security would be needed between the database and the web servers. User-specific data would now make lots of back and forth trips across the network, and would need protection. Unlike my sheltered days coding at Ask.com, I no longer have a team of brilliant network and system administrators dedicated to solving exactly these problems: masking networks, enabling access and authorization, setting up virtual clouds.  

Time for another crash course in bootstrapped engineering. 

Note: I’m also enrolled in Secure Recurring Payments 101, Amazon AutoScaling Architecture 206, and of course, the toughest one for an introvert, Business Development 342a. Thankfully I’m coming off recent successes completing my studies in Video Security and Adaptive Bitrate Tuning, as well as Fitness Video Production 101 and 102.

So that’s where I am now.  Reviewing my options for django/apache authentication methods and frameworks, SSL certificates, and the like.  Will report back when I get to the midterm (later this week!), or find a study buddy to give me a headstart.