ec2-ami-tools-1.4.0.9/0000755000000000000000000000000012050102225012721 5ustar rootrootec2-ami-tools-1.4.0.9/readme-install.txt0000644000000000000000000000562412050102225016372 0ustar rootrootAMI Tools --------- An Amazon Machine Image (AMI) is a file system image that can be mounted as a loopback device and then used as a file system for a Virtual Machine Instance. The AMI Tools are a set of command line utilities for creating, bundling and uploading AMIs to S3 storage where they can be used by EC2. The following user command line tools are provided: - ec2-bundle-image: bundle an existing AMI - ec2-bundle-vol: create an AMI from an existing installation volume and bundle it - ec2-upload-bundle: upload a bundled AMI to S3 storage - ec2-delete-bundle: delete a bundled AMI stored in S3 - ec2-download-bundle: download an existing bundle - ec2-unbundle: extract a loopback filesystem from an existing bundle - ec2-migrate-manifest: modify a manifest file for use in a different region - ec2-migrate-bundle: modify and copy an existing bundle for use in a different region Compatability ------------- The AMI Tools are available in a self contained zip file for systems that do not support RPMs. They should run on most Linux distributions provided the following dependencies are installed and available from the current search path: - curl - gzip - mkfifo - openssl - rsync - Ruby 1.8.2 or later - tee All utilies should work across Linux distributions provided the above dependencies are met. Installation ------------ To install the AMI Tools unzip the installation file into a suitable installation directory such as '/usr/local': unzip ec2-ami-tools-X.X-XXXX.zip -d This will create the directory '/ec2-ami-tools-X.X-XXXX' where X.X-XXXX represents the release and build numbers. Before running the utilities the EC2_HOME environment variable needs to be set: export EC2_HOME=/ec2-ami-tools-X.X-XXXX If you are using the EC2 API tools and cannot, or would rather not, allow them to share a home directory you can set an EC2_AMITOOL_HOME environment variable instead. This variable takes precedence if both are set. export EC2_AMITOOL_HOME=/ec2-ami-tools-X.X-XXXX The utilities can also be added to the path: export PATH=$PATH:${EC2_AMITOOL_HOME:-EC2_HOME}/bin Documentation ------------- The manual for each utility can be displayed by invoking it with the --manual parameter. For example: ec2-bundle-image --manual The help for each utility can be displayed by invoking it with the --help parameter. For example: ec2-bundle-image --help X.509 Certificates ------------------ Two X.509 certificates are required: one for EC2 and one for the user. The X.509 certificates used by the AMI Tools must be X.509 certificates and, like the private key files, must be PEM encoded. The EC2 X.509 certificate is located at: /ec2-ami-tools-X.X-XXXX/etc/ec2/amitools/cert-ec2.pem ec2-ami-tools-1.4.0.9/bin/0000755000000000000000000000000012050102225013471 5ustar rootrootec2-ami-tools-1.4.0.9/bin/ec2-upload-bundle0000775000000000000000000000027012050102225016622 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/uploadbundle.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-download-bundle0000775000000000000000000000027212050102225017147 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/downloadbundle.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-migrate-bundle0000775000000000000000000000027112050102225016767 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/migratebundle.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-unbundle0000775000000000000000000000026412050102225015706 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/unbundle.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-bundle-image0000775000000000000000000000026712050102225016426 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/bundleimage.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-migrate-manifest0000775000000000000000000000027312050102225017326 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/migratemanifest.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-delete-bundle0000775000000000000000000000027012050102225016600 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/deletebundle.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-ami-tools-version0000775000000000000000000000026712050102225017464 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/showversion.rb $* ec2-ami-tools-1.4.0.9/bin/ec2-bundle-vol0000775000000000000000000000026512050102225016142 0ustar rootroot#!/bin/bash home=${EC2_AMITOOL_HOME:-${EC2_HOME:?Neither of EC2_AMITOOL_HOME or EC2_HOME environment variables are set}} ruby -I $home/lib ${home}/lib/ec2/amitools/bundlevol.rb $* ec2-ami-tools-1.4.0.9/license.txt0000644000000000000000000001163112050102225015106 0ustar rootrootAmazon Software License This Amazon Software License ("License") governs your use, reproduction, and distribution of the accompanying software as specified below. 1. Definitions "Licensor" means any person or entity that distributes its Work. "Software" means the original work of authorship made available under this License. "Work" means the Software and any additions to or derivative works of the Software that are made available under this License. The terms "reproduce," "reproduction," "derivative works," and "distribution" have the meaning as provided under U.S. copyright law; provided, however, that for the purposes of this License, derivative works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work. Works, including the Software, are "made available" under this License by including in or with the Work either (a) a copyright notice referencing the applicability of this License to the Work, or (b) a copy of this License. 2. License Grants 2.1 Copyright Grant. Subject to the terms and conditions of this License, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free, copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense and distribute its Work and any resulting derivative works in any form. 2.2 Patent Grant. Subject to the terms and conditions of this License, each Licensor grants to you a perpetual, worldwide, non-exclusive, royalty-free patent license to make, have made, use, sell, offer for sale, import, and otherwise transfer its Work, in whole or in part. The foregoing license applies only to the patent claims licensable by Licensor that would be infringed by Licensor's Work (or portion thereof) individually and excluding any combinations with any other materials or technology. 3. Limitations 3.1 Redistribution. You may reproduce or distribute the Work only if (a) you do so under this License, (b) you include a complete copy of this License with your distribution, and (c) you retain without modification any copyright, patent, trademark, or attribution notices that are present in the Work. 3.2 Derivative Works. You may specify that additional or different terms apply to the use, reproduction, and distribution of your derivative works of the Work ("Your Terms") only if (a) Your Terms provide that the use limitation in Section 3.3 applies to your derivative works, and (b) you identify the specific derivative works that are subject to Your Terms. Notwithstanding Your Terms, this License (including the redistribution requirements in Section 3.1) will continue to apply to the Work itself. 3.3 Use Limitation. The Work and any derivative works thereof only may be used or intended for use with the web services, computing platforms or applications provided by Amazon.com, Inc. or its affiliates, including Amazon Web Services LLC. 3.4 Patent Claims. If you bring or threaten to bring a patent claim against any Licensor (including any claim, cross-claim or counterclaim in a lawsuit) to enforce any patents that you allege are infringed by any Work, then your rights under this License from such Licensor (including the grants in Sections 2.1 and 2.2) will terminate immediately. 3.5 Trademarks. This License does not grant any rights to use any Licensor's or its affiliates' names, logos, or trademarks, except as necessary to reproduce the notices described in this License. 3.6 Termination. If you violate any term of this License, then your rights under this License (including the grants in Sections 2.1 and 2.2) will terminate immediately. 4. Disclaimer of Warranty. THE WORK IS PROVIDED "AS IS" WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WARRANTIES OR CONDITIONS OF M ERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, TITLE OR NON-INFRINGEMENT. YOU BEAR THE RISK OF UNDERTAKING ANY ACTIVITIES UNDER THIS LICENSE. SOME STATES' CONSUMER LAWS DO NOT ALLOW EXCLUSION OF AN IMPLIED WARRANTY, SO THIS DISCLAIMER MAY NOT APPLY TO YOU. 5. Limitation of Liability. EXCEPT AS PROHIBITED BY APPLICABLE LAW, IN NO EVENT AND UNDER NO LEGAL THEORY, WHETHER IN TORT (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE SHALL ANY LICENSOR BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF OR RELATED TO THIS LICENSE, THE USE OR INABILITY TO USE THE WORK (INCLUDING BUT NOT LIMITED TO LOSS OF GOODWILL, BUSINESS INTERRUPTION, LOST PROFITS OR DATA, COMPUTER FAILURE OR MALFUNCTION, OR ANY OTHER COMM ERCIAL DAMAGES OR LOSSES), EVEN IF THE LICENSOR HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. Note: Other license terms may apply to certain, identified software files contained within or distributed with the accompanying software if such terms are included in the notice folder accompanying the file. Such other license terms will then apply in lieu of the terms of the Amazon Software License above. ec2-ami-tools-1.4.0.9/etc/0000755000000000000000000000000012050102225013474 5ustar rootrootec2-ami-tools-1.4.0.9/etc/ec2/0000755000000000000000000000000012050102225014145 5ustar rootrootec2-ami-tools-1.4.0.9/etc/ec2/amitools/0000755000000000000000000000000012050102225015774 5ustar rootrootec2-ami-tools-1.4.0.9/etc/ec2/amitools/cert-ec2.pem0000644000000000000000000000254312050102225020107 0ustar rootroot-----BEGIN CERTIFICATE----- MIIDzjCCAzegAwIBAgIJALDnZV+lpZdSMA0GCSqGSIb3DQEBBQUAMIGhMQswCQYD VQQGEwJaQTEVMBMGA1UECBMMV2VzdGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRv d24xJzAlBgNVBAoTHkFtYXpvbiBEZXZlbG9wbWVudCBDZW50cmUgKFNBKTEMMAoG A1UECxMDQUVTMREwDwYDVQQDEwhBRVMgVGVzdDEdMBsGCSqGSIb3DQEJARYOYWVz QGFtYXpvbi5jb20wHhcNMDUwODA5MTYwMTA5WhcNMDYwODA5MTYwMTA5WjCBoTEL MAkGA1UEBhMCWkExFTATBgNVBAgTDFdlc3Rlcm4gQ2FwZTESMBAGA1UEBxMJQ2Fw ZSBUb3duMScwJQYDVQQKEx5BbWF6b24gRGV2ZWxvcG1lbnQgQ2VudHJlIChTQSkx DDAKBgNVBAsTA0FFUzERMA8GA1UEAxMIQUVTIFRlc3QxHTAbBgkqhkiG9w0BCQEW DmFlc0BhbWF6b24uY29tMIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC8v/X5 zZv8CAVfNmvBM0br/RUcf1wU8xC5d2otFQQsQKB3qiWoj3oHeOWskOlTPFVZ8N+/ hEaMjyOUkg2+g6XEagCQtFCEBzUVoMjiQIBPiWj5CWkFtlav2zt33LZ0ErTND4xl j7FQFqbaytHU9xuQcFO2p12bdITiBs5Kwoi9bQIDAQABo4IBCjCCAQYwHQYDVR0O BBYEFPQnsX1kDVzPtX+38ACV8RhoYcw8MIHWBgNVHSMEgc4wgcuAFPQnsX1kDVzP tX+38ACV8RhoYcw8oYGnpIGkMIGhMQswCQYDVQQGEwJaQTEVMBMGA1UECBMMV2Vz dGVybiBDYXBlMRIwEAYDVQQHEwlDYXBlIFRvd24xJzAlBgNVBAoTHkFtYXpvbiBE ZXZlbG9wbWVudCBDZW50cmUgKFNBKTEMMAoGA1UECxMDQUVTMREwDwYDVQQDEwhB RVMgVGVzdDEdMBsGCSqGSIb3DQEJARYOYWVzQGFtYXpvbi5jb22CCQCw52VfpaWX UjAMBgNVHRMEBTADAQH/MA0GCSqGSIb3DQEBBQUAA4GBAJJlWll4uGlrqBzeIw7u M3RvomlxMESwGKb9gI+ZeORlnHAyZxvd9XngIcjPuU+8uc3wc10LRQUCn45a5hFs zaCp9BSewLCCirn6awZn2tP8JlagSbjrN9YShStt8S3S/Jj+eBoRvc7jJnmEeMkx O0wHOzp5ZHRDK7tGULD6jCfU -----END CERTIFICATE----- ec2-ami-tools-1.4.0.9/etc/ec2/amitools/mappings.csv0000644000000000000000000000116112050102225020326 0ustar rootrooteu-west-1,us-east-1,manifest ari-6b0d251f,ari-a23adfcb,initrd-2.6.20-1.3002.fc6xen.ari.manifest.xml ari-7f0d250b,ari-b31cf9da,ec2-initrd-2.6.21.7-2.fc8xen.x86_64.manifest.xml aki-6a0d251e,aki-a53adfcc,vmlinuz-2.6.20-1.3002.fc6xen.aki.manifest.xml aki-550d2521,aki-9800e5f1,vmlinuz-2.6.18-xenU-ec2-v1.0.x86_64.aki.manifest.xml aki-780d250c,aki-b51cf9dc,ec2-vmlinuz-2.6.21.7-2.fc8xen.x86_64.manifest.xml ari-7d0d2509,ari-a51cf9cc,ec2-initrd-2.6.21.7-2.fc8xen.i386.manifest.xml aki-540d2520,aki-9b00e5f2,vmlinuz-2.6.18-xenU-ec2-v1.0.i386.aki.manifest.xml aki-7e0d250a,aki-a71cf9ce,ec2-vmlinuz-2.6.21.7-2.fc8xen.i386.manifest.xmlec2-ami-tools-1.4.0.9/etc/ec2/amitools/cert-ec2-gov.pem0000644000000000000000000000176512050102225020705 0ustar rootroot-----BEGIN CERTIFICATE----- MIICvzCCAigCCQD3V6lFvX6dzDANBgkqhkiG9w0BAQUFADCBozELMAkGA1UEBhMC VVMxCzAJBgNVBAgTAldBMRAwDgYDVQQHEwdTZWF0dGxlMRMwEQYDVQQKEwpBbWF6 b24uY29tMRYwFAYDVQQLEw1FQzIgQXV0aG9yaXR5MRowGAYDVQQDExFFQzIgQU1J IEF1dGhvcml0eTEsMCoGCSqGSIb3DQEJARYdZWMyLWFtaS1nb3Ytd2VzdC0xQGFt YXpvbi5jb20wHhcNMTEwODEyMTcyNjE1WhcNMjEwODA5MTcyNjE1WjCBozELMAkG A1UEBhMCVVMxCzAJBgNVBAgTAldBMRAwDgYDVQQHEwdTZWF0dGxlMRMwEQYDVQQK EwpBbWF6b24uY29tMRYwFAYDVQQLEw1FQzIgQXV0aG9yaXR5MRowGAYDVQQDExFF QzIgQU1JIEF1dGhvcml0eTEsMCoGCSqGSIb3DQEJARYdZWMyLWFtaS1nb3Ytd2Vz dC0xQGFtYXpvbi5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBANshKnhw DUZ2/6VJwVTsXMUI1CGd5rpSpSLUCHGuqII+BDUvnp/sPxd1u6+I1QrbaaBAOm6+ evM77M7vNJXY3+JW00VOs9NgPEXBmn6UV4R1P7DljKurWGmRp8Fj1yVU4sSgZqqv 74SyhD0Z4ASczVcOiTZICeuQoJwmeZ8F20oLAgMBAAEwDQYJKoZIhvcNAQEFBQAD gYEAH3vpkD80ngP1e18UYSVBCODArik+aeUPAzJrPDYorrnffbamks50IMTktyiu za1JuplrvVsAKcQhyoPOq69bwRDg4L8VOXSCjjvsNuEhHL603h8jn6ghEouPCPl7 8s4Sr5XikmAgwFcPb/frNnLuZsSol08tISgPOlFg4KLv/bo= -----END CERTIFICATE----- ec2-ami-tools-1.4.0.9/lib/0000755000000000000000000000000012050102224013466 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/0000755000000000000000000000000012050102225014140 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/oem/0000755000000000000000000000000012050102225014720 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/oem/LICENSE.txt0000644000000000000000000000502612050102225016546 0ustar rootrootRuby is copyrighted free software by Yukihiro Matsumoto . You can redistribute it and/or modify it under either the terms of the GPL (see COPYING.txt file), or the conditions below: 1. You may make and give away verbatim copies of the source form of the software without restriction, provided that you duplicate all of the original copyright notices and associated disclaimers. 2. You may modify your copy of the software in any way, provided that you do at least ONE of the following: a) place your modifications in the Public Domain or otherwise make them Freely Available, such as by posting said modifications to Usenet or an equivalent medium, or by allowing the author to include your modifications in the software. b) use the modified software only within your corporation or organization. c) rename any non-standard executables so the names do not conflict with standard executables, which must also be provided. d) make other distribution arrangements with the author. 3. You may distribute the software in object code or executable form, provided that you do at least ONE of the following: a) distribute the executables and library files of the software, together with instructions (in the manual page or equivalent) on where to get the original distribution. b) accompany the distribution with the machine-readable source of the software. c) give non-standard executables non-standard names, with instructions on where to get the original software distribution. d) make other distribution arrangements with the author. 4. You may modify and include the part of the software into any other software (possibly commercial). But some files in the distribution are not written by the author, so that they are not under this terms. They are gc.c(partly), utils.c(partly), regex.[ch], st.[ch] and some files under the ./missing directory. See each file for the copying condition. 5. The scripts and library files supplied as input to or produced as output from the software do not automatically fall under the copyright of the software, but belong to whomever generated them, and may be sold commercially, and may be aggregated with this software. 6. THIS SOFTWARE IS PROVIDED "AS IS" AND WITHOUT ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. ec2-ami-tools-1.4.0.9/lib/ec2/oem/open4.rb0000644000000000000000000002142212050102225016273 0ustar rootroot# ------------------------------------------------------------------------ # Name: Open4 # Version: 4-0.9.6 # Author: Ara T. Howard # License: Ruby License # ------------------------------------------------------------------------ # vim: ts=2:sw=2:sts=2:et:fdm=marker require 'fcntl' require 'timeout' require 'thread' module Open4 #--{{{ VERSION = '0.9.6' def self.version() VERSION end class Error < ::StandardError; end def popen4(*cmd, &b) #--{{{ pw, pr, pe, ps = IO.pipe, IO.pipe, IO.pipe, IO.pipe verbose = $VERBOSE begin $VERBOSE = nil ps.last.fcntl(Fcntl::F_SETFD, Fcntl::FD_CLOEXEC) cid = fork { pw.last.close STDIN.reopen pw.first pw.first.close pr.first.close STDOUT.reopen pr.last pr.last.close pe.first.close STDERR.reopen pe.last pe.last.close STDOUT.sync = STDERR.sync = true begin exec(*cmd) raise 'forty-two' rescue Exception => e Marshal.dump(e, ps.last) ps.last.flush end ps.last.close unless (ps.last.closed?) exit! } ensure $VERBOSE = verbose end [pw.first, pr.last, pe.last, ps.last].each{|fd| fd.close} begin e = Marshal.load ps.first raise(Exception === e ? e : "unknown failure!") rescue EOFError # If we get an EOF error, then the exec was successful 42 ensure ps.first.close end pw.last.sync = true pi = [pw.last, pr.first, pe.first] if b begin b[cid, *pi] Process.waitpid2(cid).last ensure pi.each{|fd| fd.close unless fd.closed?} end else [cid, pw.last, pr.first, pe.first] end #--}}} end alias open4 popen4 module_function :popen4 module_function :open4 class SpawnError < Error #--{{{ attr 'cmd' attr 'status' attr 'signals' def exitstatus @status.exitstatus end def initialize cmd, status @cmd, @status = cmd, status @signals = {} if status.signaled? @signals['termsig'] = status.termsig @signals['stopsig'] = status.stopsig end sigs = @signals.map{|k,v| "#{ k }:#{ v.inspect }"}.join(' ') super "cmd <#{ cmd }> failed with status <#{ exitstatus.inspect }> signals <#{ sigs }>" end #--}}} end class ThreadEnsemble #--{{{ attr 'threads' def initialize cid @cid, @threads, @argv, @done, @running = cid, [], [], Queue.new, false @killed = false end def add_thread *a, &b @running ? raise : (@argv << [a, b]) end # # take down process more nicely # def killall c = Thread.critical return nil if @killed Thread.critical = true (@threads - [Thread.current]).each{|t| t.kill rescue nil} @killed = true ensure Thread.critical = c end def run @running = true begin @argv.each do |a, b| @threads << Thread.new(*a) do |*a| begin b[*a] ensure killall rescue nil if $! @done.push Thread.current end end end rescue killall raise ensure all_done end @threads.map{|t| t.value} end def all_done @threads.size.times{ @done.pop } end #--}}} end def to timeout = nil #--{{{ Timeout.timeout(timeout){ yield } #--}}} end module_function :to def new_thread *a, &b #--{{{ cur = Thread.current Thread.new(*a) do |*a| begin b[*a] rescue Exception => e cur.raise e end end #--}}} end module_function :new_thread def getopts opts = {} #--{{{ lambda do |*args| keys, default, ignored = args catch('opt') do [keys].flatten.each do |key| [key, key.to_s, key.to_s.intern].each do |key| throw 'opt', opts[key] if opts.has_key?(key) end end default end end #--}}} end module_function :getopts def relay src, dst = nil, t = nil #--{{{ unless src.nil? if src.respond_to? :gets while buf = to(t){ src.gets } dst << buf if dst end elsif src.respond_to? :each q = Queue.new th = nil timer_set = lambda do |t| th = new_thread{ to(t){ q.pop } } end timer_cancel = lambda do |t| th.kill if th rescue nil end timer_set[t] begin src.each do |buf| timer_cancel[t] dst << buf if dst timer_set[t] end ensure timer_cancel[t] end elsif src.respond_to? :read buf = to(t){ src.read } dst << buf if dst else buf = to(t){ src.to_s } dst << buf if dst end end #--}}} end module_function :relay def spawn arg, *argv #--{{{ argv.unshift(arg) opts = ((argv.size > 1 and Hash === argv.last) ? argv.pop : {}) argv.flatten! cmd = argv.join(' ') getopt = getopts opts ignore_exit_failure = getopt[ 'ignore_exit_failure', getopt['quiet', false] ] ignore_exec_failure = getopt[ 'ignore_exec_failure', !getopt['raise', true] ] exitstatus = getopt[ %w( exitstatus exit_status status ) ] stdin = getopt[ %w( stdin in i 0 ) << 0 ] stdout = getopt[ %w( stdout out o 1 ) << 1 ] stderr = getopt[ %w( stderr err e 2 ) << 2 ] pid = getopt[ 'pid' ] timeout = getopt[ %w( timeout spawn_timeout ) ] stdin_timeout = getopt[ %w( stdin_timeout ) ] stdout_timeout = getopt[ %w( stdout_timeout io_timeout ) ] stderr_timeout = getopt[ %w( stderr_timeout ) ] status = getopt[ %w( status ) ] cwd = getopt[ %w( cwd dir ) ] exitstatus = case exitstatus when TrueClass, FalseClass ignore_exit_failure = true if exitstatus [0] else [*(exitstatus || 0)].map{|i| Integer i} end stdin ||= '' if stdin_timeout stdout ||= '' if stdout_timeout stderr ||= '' if stderr_timeout started = false status = begin chdir(cwd) do Timeout::timeout(timeout) do popen4(*argv) do |c, i, o, e| started = true %w( replace pid= << push update ).each do |msg| break(pid.send(msg, c)) if pid.respond_to? msg end te = ThreadEnsemble.new c te.add_thread(i, stdin) do |i, stdin| relay stdin, i, stdin_timeout i.close rescue nil end te.add_thread(o, stdout) do |o, stdout| relay o, stdout, stdout_timeout end te.add_thread(e, stderr) do |o, stderr| relay e, stderr, stderr_timeout end te.run end end end rescue raise unless(not started and ignore_exec_failure) end raise SpawnError.new(cmd, status) unless (ignore_exit_failure or (status.nil? and ignore_exec_failure) or exitstatus.include?(status.exitstatus)) status #--}}} end module_function :spawn def chdir cwd, &block return(block.call Dir.pwd) unless cwd Dir.chdir cwd, &block end module_function :chdir def background arg, *argv #--{{{ require 'thread' q = Queue.new opts = { 'pid' => q, :pid => q } case argv.last when Hash argv.last.update opts else argv.push opts end thread = Thread.new(arg, argv){|arg, argv| spawn arg, *argv} sc = class << thread; self; end sc.module_eval { define_method(:pid){ @pid ||= q.pop } define_method(:spawn_status){ @spawn_status ||= value } define_method(:exitstatus){ @exitstatus ||= spawn_status.exitstatus } } thread #--}}} end alias bg background module_function :background module_function :bg def maim pid, opts = {} #--{{{ getopt = getopts opts sigs = getopt[ 'signals', %w(SIGTERM SIGQUIT SIGKILL) ] suspend = getopt[ 'suspend', 4 ] pid = Integer pid existed = false sigs.each do |sig| begin Process.kill sig, pid existed = true rescue Errno::ESRCH return(existed ? nil : true) end return true unless alive? pid sleep suspend return true unless alive? pid end return(not alive?(pid)) #--}}} end module_function :maim def alive pid #--{{{ pid = Integer pid begin Process.kill 0, pid true rescue Errno::ESRCH false end #--}}} end alias alive? alive module_function :alive module_function :'alive?' #--}}} end def open4(*cmd, &b) cmd.size == 0 ? Open4 : Open4::popen4(*cmd, &b) end ec2-ami-tools-1.4.0.9/lib/ec2/platform.rb0000644000000000000000000000415312050102225016314 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # Wrapper around RUBY_PLATFORM module EC2 module Platform IMPLEMENTATIONS = [ [/darwin/i, :unix, :macosx ], [/linux/i, :unix, :linux ], [/freebsd/i, :unix, :freebsd], [/netbsd/i, :unix, :netbsd ], [/solaris/i, :unix, :solaris], [/irix/i, :unix, :irix ], [/cygwin/i, :unix, :cygwin ], [/mswin/i, :win32, :mswin ], [/mingw/i, :win32, :mingw ], [/bccwin/i, :win32, :bccwin ], [/wince/i, :win32, :wince ], [/vms/i, :vms, :vms ], [/os2/i, :os2, :os2 ], [nil, :unknown, :unknown], ] ARCHITECTURES = [ [/(i\d86)/i, :i386 ], [/x86_64/i, :x86_64 ], [/ia64/i, :ia64 ], [/alpha/i, :alpha ], [/sparc/i, :sparc ], [/mips/i, :mips ], [/powerpc/i, :powerpc ], [nil, :unknown ], ] def self.guess os = :unknown impl = :unknown arch = :unknown IMPLEMENTATIONS.each do |r, o, i| if r and RUBY_PLATFORM =~ r os, impl = [o, i] break end end ARCHITECTURES.each do |r, a| if r and RUBY_PLATFORM =~ r arch = a break end end return [os, impl, arch] end OS, IMPL, ARCH = guess end end if __FILE__ == $0 include EC2::Platform puts "Platform OS=#{Platform::OS}, IMPL=#{Platform::IMPL}, ARCH=#{Platform::ARCH}" end ec2-ami-tools-1.4.0.9/lib/ec2/common/0000755000000000000000000000000012050102225015430 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/common/curl.rb0000644000000000000000000000773112050102225016732 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # ----------------------------------------------------------------------------- # This is a light ruby wrapper around the curl command-line utility. # Unless the run with the -f/--fail flag, the curl utility returns an exit-code # of value 0 for requests that produce responses with HTTP errors codes greater # than or equal to 400; it returns 22 and swallows the response. This wrapper # attempts to emulate the -f/--fail flag while making available response codes # with minimal verbosity. # ----------------------------------------------------------------------------- require 'ec2/oem/open4' require 'tmpdir' module EC2 module Common module Curl class Error < RuntimeError attr_reader :code def initialize(message, code=22) super message @code = code end end class Response attr_reader :code, :type def initialize(code, type) @code = code.to_i @type = type end def success? (200..299).include? @code end def redirect? (300..399).include? @code end def text? @type =~ /(text|xml)/ end end class Result attr_reader :stdout, :stderr, :status, :response def initialize(stdout, stderr, status, response = nil) unless response.nil? or response.is_a? EC2::Common::Curl::Response raise ArgumentError.new('invalid response argument') end @stdout = stdout @stderr = stderr @status = status @response= response end def success? @status == 0 end end def self.invoke(command, debug=false) invocation = "curl -sSL #{command}" # invocation = "curl -vsSL #{command}" if debug invocation << ' -w "Response-Code: %{http_code}\nContent-Type: %{content_type}"' STDERR.puts invocation if debug pid, stdin, stdout, stderr = Open4::popen4(invocation) pid, status = Process::waitpid2 pid [status, stdout, stderr] end def self.print_error(command, status, out, err) puts "----COMMAND------------------" puts command puts "----EXIT-CODE----------------" puts status.exitstatus.inspect puts "----STDOUT-------------------" puts out puts "----STDERR-------------------" puts err puts "-----------------------------" end def self.execute(command, debug = false) status, stdout, stderr = self.invoke(command, debug) out = stdout.read err = stderr.read if status.exitstatus == 0 code, type = out.chomp.split("\n").zip(['Response-Code', 'Content-Type']).map do |line, name| (m = Regexp.new("^#{name}: (\\S+)$").match(line.chomp)) ? m.captures[0] : nil end if code.nil? self.print_error(command, status, out, err) if debug raise EC2::Common::Curl::Error.new( 'Invalid curl output for response-code. Is the server up and reachable?' ) end response = EC2::Common::Curl::Response.new(code, type) return EC2::Common::Curl::Result.new(out, err, status.exitstatus, response) else self.print_error(command, status, out, err) if debug return EC2::Common::Curl::Result.new(out, err, status.exitstatus) end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/common/http.rb0000644000000000000000000003603012050102225016736 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # Module that provides http functionality. # --------------------------------------------------------------------------- require 'uri' require 'set' require 'time' require 'base64' require 'tmpdir' require 'tempfile' require 'fileutils' require 'ec2/common/curl' require 'ec2/amitools/crypto' module EC2 module Common module HTTP class Response < EC2::Common::Curl::Response attr_reader :body def initialize(code, type, body=nil) super code, type @body = body end end class Headers MANDATORY = ['content-md5', 'content-type', 'date'] # Order matters. X_AMZ_PREFIX = 'x-amz' X_AMZ_SECURITY_TOKEN = 'x-amz-security-token' def initialize(verb) raise ArgumentError.new('invalid verb') if verb.to_s.empty? @headers = {} @verb = verb end #--------------------------------------------------------------------- # Add a Header key-value pair. def add(name, value) raise ArgumentError.new("name '#{name.inspect}' must be a String") unless name.is_a? String raise ArgumentError.new("value '#{value.inspect}' (#{name}) must be a String") unless value.is_a? String @headers[name.downcase.strip] = value.strip end #----------------------------------------------------------------------- # Sign the headers using HMAC SHA1. def sign(creds, aws_secret_access_key, url, bucket) aws_access_key_id = nil delegation_token = nil if (creds.is_a?(Hash)) aws_access_key_id = creds['aws_access_key_id'] aws_secret_access_key = creds['aws_secret_access_key'] delegation_token = creds['aws_delegation_token'] end @headers['date'] = Time.now.httpdate # add HTTP Date header # Build the data to sign. data = @verb + "\n" # Deal with mandatory headers first: MANDATORY.each do |name| data += @headers.fetch(name, "") + "\n" end unless delegation_token.nil? @headers[X_AMZ_SECURITY_TOKEN] = delegation_token end # Add mandatory headers and those that start with the x-amz prefix. @headers.sort.each do |name, value| # Headers that start with x-amz must have both their name and value # added. if name =~ /^#{X_AMZ_PREFIX}/ data += name + ":" + value +"\n" end end # Ignore everything in the URL after the question mark unless, by the # S3 protocol, it signifies an acl, torrent, logging or location parameter unless bucket.nil? or bucket.size == 0 data << "/#{bucket}" end data << URI.parse(url).path ['acl', 'logging', 'torrent', 'location'].each do |item| regex = Regexp.new("[&?]#{item}($|&|=)") data << '?' + item if regex.match(url) end # Sign headers and then put signature back into headers. signature = Base64.encode64(Crypto::hmac_sha1(aws_secret_access_key, data)) signature.chomp! @headers['Authorization'] = "AWS #{aws_access_key_id}:#{signature}" end #----------------------------------------------------------------------- # Return the headers as a map from header name to header value. def get return @headers.clone end #----------------------------------------------------------------------- # Return the headers as curl arguments of the form "-H name:value". def curl_arguments @headers.map { | name, value | "-H \"#{name}:#{value}\""}.join(' ') end end #----------------------------------------------------------------------- # Errors. class Error < RuntimeError attr_reader :code def initialize(msg, code = nil) super(msg) @code = code || 1 end class PathInvalid < Error; end class Write < Error; end class BadDigest < Error def initialize(file, expected, obtained) super("Digest for file '#{file}' #{obtained} differs from expected digest #{digest}") end end class Transfer < Error; end class Retrieve < Transfer; end class Redirect < Error attr_accessor :code, :endpoint def initialize(code, endpoint) super("Redirected (#{code}) to new endpoint: #{endpoint}") @code = code @endpoint = endpoint end end end #----------------------------------------------------------------------- # Invoke curl with arguments given and process results of output file def HTTP::invoke(url, arguments, outfile, debug=false) begin raise ArgumentError.new(outfile) unless File.exists? outfile result = EC2::Common::Curl.execute("#{arguments.join(' ')} '#{url}'", debug) if result.success? if result.response.success? return result.response else synopsis= 'Server.Error(' + result.response.code.to_s + '): ' message = result.stderr + ' ' if result.response.type == 'application/xml' require 'rexml/document' doc = REXML::Document.new(IO.read(outfile)) if doc.root if result.response.redirect? endpoint = REXML::XPath.first(doc, '/Error/Endpoint').text raise Error::Redirect.new(result.response.code, endpoint) end content = REXML::XPath.first(doc, '/Error/Code') unless content.nil? or content.text.empty? synopsis= 'Server.'+ content.text + '(' + result.response.code.to_s + '): ' end content = REXML::XPath.first(doc, '/Error/Message') message = content.text unless content.nil? end else if result.response.type =~ /text/ message << IO.read(outfile) end end raise Error::Transfer.new(synopsis + message, result.response.code) end else synopsis= 'Curl.Error(' + result.status.to_s + '): ' message = result.stderr.split("\n").map { |line| if (m = /^curl:\s+(?:\(\d{1,2}\)\s+)*(.*)$/.match(line)) (m.captures[0] == "try 'curl --help' for more information") ? '' : m.captures[0] else line.strip end }.join("\n") output = result.stdout.chomp if debug and not output.empty? message << "\nCurl.Output: " + output.gsub("\n", "\nCurl.Output: ") end raise Error::Transfer.new(synopsis + message.strip + '.', result.status) end rescue EC2::Common::Curl::Error => e raise Error::Transfer.new(e.message, e.code) end end #----------------------------------------------------------------------- # Delete the file at the specified url. def HTTP::delete(url, bucket, options={}, user=nil, pass=nil, debug=false) raise ArgumentError.new('Bad options in HTTP::delete') unless options.is_a? Hash begin output = Tempfile.new('ec2-delete-response') output.close arguments = ['-X DELETE'] headers = EC2::Common::HTTP::Headers.new('DELETE') options.each do |name, value| headers.add(name, value) end headers.sign(user, pass, url, bucket) if user and pass arguments << headers.curl_arguments arguments << '-o ' + output.path response = HTTP::invoke(url, arguments, output.path, debug) return EC2::Common::HTTP::Response.new(response.code, response.type) ensure output.close(true) GC.start end end #----------------------------------------------------------------------- # Put the file at the specified path to the specified url. The content of # the options hash will be passed as HTTP headers. If the username and # password options are specified, then the headers will be signed. def HTTP::put(url, bucket, path, options={}, user=nil, pass=nil, debug=false) path ||= "/dev/null" raise Error::PathInvalid.new(path) unless path and File::exist?(path) raise ArgumentError.new('Bad options in HTTP::put') unless options.is_a? Hash begin output = Tempfile.new('ec2-put-response') output.close arguments = [] headers = EC2::Common::HTTP::Headers.new('PUT') options.each do |name, value| headers.add(name, value) end headers.sign(user, pass, url, bucket) if user and pass arguments << headers.curl_arguments arguments << '-T ' + path arguments << '-o ' + output.path response = HTTP::invoke(url, arguments, output.path, debug) return EC2::Common::HTTP::Response.new(response.code, response.type) ensure output.close(true) GC.start end end #----------------------------------------------------------------------- # Put the file at the specified path to the specified url. The content of # the options hash will be passed as HTTP headers. If the username and # password options are specified, then the headers will be signed. def HTTP::putdir(url, bucket, path, options={}, user=nil, pass=nil, debug=false) raise Error::PathInvalid.new(path) unless path and File::exist?(path) raise ArgumentError.new('Bad options in HTTP::putdir') unless options.is_a? Hash begin output = Tempfile.new('ec2-put-response') output.close arguments = [] headers = EC2::Common::HTTP::Headers.new('PUT') options.each do |name, value| headers.add(name, value) end headers.sign(user, pass, url, bucket) if user and pass arguments << headers.curl_arguments arguments << '-T - ' arguments << '-o ' + output.path arguments << " < #{path}" response = HTTP::invoke(url, arguments, output.path, debug) return EC2::Common::HTTP::Response.new(response.code, response.type) ensure output.close(true) GC.start end end #----------------------------------------------------------------------- # Save the file at specified url, to the local file at specified path. # The local file will be created if necessary, or overwritten already # existing. If specified, the expected digest is compare to that of the # retrieved file which gets deleted if the calculated digest does not meet # expectations. If no path is specified, and the response is a 200 OK, the # content of the response will be returned as a String def HTTP::get(url, bucket, path=nil, options={}, user=nil, pass=nil, size=nil, digest=nil, debug=false) raise ArgumentError.new('Bad options in HTTP::get') unless options.is_a? Hash arguments = [] buffer = nil if path.nil? buffer = Tempfile.new('ec2-get-response') buffer.close path = buffer.path else directory = File.dirname(path) FileUtils.mkdir_p(directory) unless File.exist?(directory) end headers = EC2::Common::HTTP::Headers.new('GET') options.each do |name, value| headers.add(name, value) end headers.sign(user, pass, url, bucket) if user and pass arguments << headers.curl_arguments arguments << "--max-filesize #{size}" if size arguments << '-o ' + path begin FileUtils.touch path response = HTTP::invoke(url, arguments, path, debug) body = nil if response.success? if digest obtained = IO.popen("openssl sha1 #{path}") { |io| io.readline.split(/\s+/).last.strip } unless digest == obtained File.delete(path) if File.exists?(path) and not buffer.is_a? Tempfile raise Error::BadDigest.new(path, expected, obtained) end end if buffer.is_a? Tempfile buffer.open; body = buffer.read end else File.delete(path) if File.exist?(path) and not buffer.is_a? Tempfile end return EC2::Common::HTTP::Response.new(response.code, response.type, body) rescue Error::Transfer => e File::delete(path) if File::exist?(path) and not buffer.is_a? Tempfile raise Error::Retrieve.new(e.message, e.code) ensure if buffer.is_a? Tempfile buffer.close buffer.unlink if File.exists? path GC.start end end end #----------------------------------------------------------------------- # Get the HEAD response for the specified url. def HTTP::head(url, bucket, options={}, user=nil, pass=nil, debug=false) raise ArgumentError.new('Bad options in HTTP::head') unless options.is_a? Hash begin output = Tempfile.new('ec2-head-response') output.close arguments = ['--head'] headers = EC2::Common::HTTP::Headers.new('HEAD') options.each do |name, value| headers.add(name, value) end headers.sign(user, pass, url, bucket) if user and pass arguments << headers.curl_arguments arguments << '-o ' + output.path response = HTTP::invoke(url, arguments, output.path, debug) return EC2::Common::HTTP::Response.new(response.code, response.type) rescue Error::Transfer => e raise Error::Retrieve.new(e.message, e.code) ensure output.close(true) GC.start end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/common/s3support.rb0000644000000000000000000001400712050102225017741 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # --------------------------------------------------------------------------- # Module that provides higher-level S3 functionality # --------------------------------------------------------------------------- require 'cgi' require 'ec2/common/http' module EC2 module Common class S3Support attr_accessor :s3_url, :user, :pass # Return true if the bucket name is S3 safe. # # Per the S3 dev guide @ http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?BucketRestrictions.html # - Be between 3 and 255 characters long # - Start with a number or letter # - Contain lowercase letters, numbers, periods (.), underscores (_), and dashes (-) # - Not be in an IP address style (e.g., "192.168.5.4") # # * Notes: # - !!(....) silliness to force a boolean to be returned def self.bucket_name_s3_safe?(bucket_name) # Can most probably fold this all into 1 grand regexp but # for now opt for semi-clarity. !!((3..255).include?(bucket_name.length) and (/^[a-z0-9][a-z0-9\._-]+$/ =~ bucket_name) and (/^(\d{1,3}\.){3}\d{1,3}$/ !~ bucket_name)) end # Return true if the bucket name is S3 (v2) safe. # # Per the S3 dev guide @ http://docs.amazonwebservices.com/AmazonS3/2006-03-01/index.html?BucketRestrictions.html # - Bucket names should not contain underscores (_) # - Bucket names should be between 3 and 63 characters long # - Bucket names should not end with a dash # - Bucket names cannot contain dashes next to periods (e.g., "my-.bucket.com" and "my.-bucket" are invalid) # - Bucket names must only contain lower case letters # # and naturally also fulfills bucket_name_s3_safe?() requirements # # * Notes: # - !!(....) silliness to force a boolean to be returned def self.bucket_name_s3_v2_safe?(bucket_name) # Can most probably fold this all into 1 grand regexp but # for now opt for semi-clarity. !!(self.bucket_name_s3_safe?(bucket_name) and bucket_name.length <= 63 and not bucket_name.include?('_') and bucket_name[-1] != ?- and /(-\.)|(\.-)/ !~ bucket_name) end def initialize(s3_url, user, pass, format=nil, debug=nil) @user = user @pass = pass @s3_url = fix_s3_url(s3_url) @format = format || :subdomain @debug = debug end def fix_s3_url(s3_url) if s3_url !~ %r{://} s3_url = "https://#{s3_url}" end if s3_url[-1..-1] != "/" s3_url << "/" end s3_url end def get_bucket_url(bucket) case @format when :subdomain protocol, base_domain = @s3_url.split("://") return ["#{protocol}://#{bucket}.#{base_domain}", bucket] when :path return ["#{@s3_url}#{bucket}/", nil] end end def get_acl(bucket, key, options={}) begin url, bkt = get_bucket_url(bucket) url << CGI::escape(key) + '?acl' return EC2::Common::HTTP::get(url, bkt, nil, options, @user, @pass, nil, nil, @debug) end end def check_bucket_exists(bucket, options={}) url, bkt = get_bucket_url(bucket) return EC2::Common::HTTP::head(url, bkt, options, @user, @pass, @debug) end def get_bucket_location(bucket, options={}) url, bkt = get_bucket_url(bucket) url << "?location" return EC2::Common::HTTP::get(url, bkt, nil, options, @user, @pass, nil, nil, @debug) end def create_bucket(bucket, location, options={}) url, bkt = get_bucket_url(bucket) begin buffer = Tempfile.new('ec2-create-bucket') if (location != nil) buffer.write("#{location}") end buffer.close return EC2::Common::HTTP::putdir(url, bkt, buffer.path, options, @user, @pass, @debug) ensure buffer.unlink end end def list_bucket(bucket, prefix=nil, max_keys=nil, path=nil, options={}) url, bkt = get_bucket_url(bucket) params = [] params << "prefix=#{CGI::escape(prefix)}" if prefix params << "max-keys=#{CGI::escape(max_keys)}" if max_keys url << "?" + params.join("&") unless params.empty? return EC2::Common::HTTP::get(url, bkt, path, options, @user, @pass, nil, nil, @debug) end def get(bucket, key, path=nil, options={}) url, bkt = get_bucket_url(bucket) url << CGI::escape(key) return EC2::Common::HTTP::get(url, bkt, path, options, @user, @pass, nil, nil, @debug) end def put(bucket, key, file, options={}) url, bkt = get_bucket_url(bucket) url << CGI::escape(key) return EC2::Common::HTTP::put(url, bkt, file, options, @user, @pass, @debug) end def copy(bucket, key, source, options={}) url, bkt = get_bucket_url(bucket) url << CGI::escape(key) options['x-amz-copy-source'] = CGI::escape(source) return EC2::Common::HTTP::put(url, bkt, nil, options, @user, @pass, @debug) end def delete(bucket, key="", options={}) url, bkt = get_bucket_url(bucket) url << CGI::escape(key) return EC2::Common::HTTP::delete(url, bkt, options, @user, @pass, @debug) end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/0000755000000000000000000000000012050102225015764 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris.rb0000644000000000000000000000263212050102225017770 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ require 'ec2/platform/base' require 'ec2/platform/solaris/identity' require 'ec2/platform/solaris/architecture' require 'ec2/platform/solaris/fstab' require 'ec2/platform/solaris/mtab' require 'ec2/platform/solaris/image' require 'ec2/platform/solaris/rsync' require 'ec2/platform/solaris/tar' require 'ec2/platform/solaris/uname' require 'ec2/platform/solaris/pipeline' require 'ec2/platform/solaris/constants' module EC2 module Platform module Solaris class System < EC2::Platform::Base::System BUNDLING_ARCHITECTURE = EC2::Platform::Solaris::Architecture.bundling #---------------------------------------------------------------------# def self.superuser? return `id -u`.strip == '0' end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/0000755000000000000000000000000012050102225017440 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/mtab.rb0000644000000000000000000000200312050102225020703 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/platform/linux/mtab' module EC2 module Platform module Solaris LOCAL_FS_TYPES = [ 'ext2', 'ext3', 'xfs', 'jfs', 'reiserfs', 'tmpfs', 'ufs', 'sharefs', 'dev', 'devfs', 'ctfs', 'mntfs', 'proc', 'lofs', 'objfs', 'fd', 'autofs' ] class Mtab < EC2::Platform::Linux::Mtab LOCATION = '/etc/mnttab' def initialize(filename = LOCATION) super filename end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/fstab.rb0000644000000000000000000000344612050102225021073 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/platform/linux/fstab' module EC2 module Platform module Solaris class Fstab < EC2::Platform::Linux::Fstab LOCATION = '/etc/vfstab' def initialize(filename = LOCATION) super filename end DEFAULT = IO.read(File.join('/etc', 'vfstab')) rescue < #{file.path}" end command + list.join(' & ') end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/tar.rb0000644000000000000000000000232412050102225020554 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/platform/linux/tar' require 'ec2/platform/solaris/constants' module EC2 module Platform module Solaris class Tar < EC2::Platform::Linux::Tar class Command < EC2::Platform::Linux::Tar::Command EXECUTABLE=EC2::Platform::Solaris::Constants::Utility::TAR def initialize(e=EXECUTABLE) super e end end class Version < EC2::Platform::Linux::Tar::Version def default s = `#{Command.new.version.expand}`.strip s = nil unless $? == 0 s end def self.current Version.new end end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/constants.rb0000644000000000000000000000214112050102225021777 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------ # Solaris overrides for constants go here #------------------------------------------------------------------------ require 'ec2/platform/base/constants' module EC2 module Platform module Solaris module Constants module Bundling include EC2::Platform::Base::Constants::Bundling DESTINATION = '/mnt' end module Utility OPENSSL = '/usr/sfw/bin/openssl' TAR = '/usr/sfw/bin/gtar' end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/image.rb0000644000000000000000000003171012050102225021051 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #---------------------------------------------------------------------# # Create a Solaris EC2 Image as follows: # - create a bootable file-system archive in the SUN flash format # - create, format and mount a blank file-system image # - replicate the archive section of the flash archive into the image # - customize the image # # Fasten your seat-belts and grab a pillow; this is painfully slow. # Initial tests show an average bundling time of a virgin OpenSolaris # system using this algorithm to be about 85-90 minutes. Optimization # involves rewriting flar to combine the "flar create" and "flar split" # steps into a "flar replicate" step #---------------------------------------------------------------------# require 'fileutils' require 'ec2/oem/open4' require 'ec2/amitools/fileutil' require 'ec2/amitools/syschecks' require 'ec2/amitools/exception' require 'ec2/platform/solaris/mtab' require 'ec2/platform/solaris/fstab' require 'ec2/platform/solaris/constants' module EC2 module Platform module Solaris class ExecutionError < RuntimeError end # This class encapsulate functionality to create an file loopback image # from a volume. The image is created using mkfile. Sub-directories of the # volume, including mounts of local filesystems, are copied to the image. # Symbolic links are preserved wherever possible. class Image EXCLUDES = [ '/mnt' ] WORKSPACE = '/mnt/ec2-bundle-workspace' MOUNT = File.join( WORKSPACE, 'mnt' ) ARCHIVE = File.join( WORKSPACE, 'archive' ) PROFILING = true RETRIES = 5 DELAY = 10 #---------------------------------------------------------------------# def initialize( volume, # path to volume to be bundled filename, # name of image file to create size, # size of image file in MB exclude, # list of directories to exclude includes, # This does absolutely nothing on solaris - warrenr filter, # Same as above - warrenr vfstab=nil, # file system table to use debug = false ) @volume = volume @filename = filename @size = size @exclude = exclude @debug = debug if vfstab.nil? or vfstab == :legacy @vfstab = EC2::Platform::Solaris::Fstab::DEFAULT elsif File.exists? vfstab @vfstab = IO.read(vfstab) else @vfstab = vfstab end # Exclude the workspace if it is in the volume being bundled. @exclude << WORKSPACE if( WORKSPACE.index(volume) == 0 ) end #---------------------------------------------------------------------# # Clone a running volume into a bootable Amazon Machine Image. def make begin announce( "Cloning #{@volume} into image file #{@filename}...", true) announce( 'Excluding: ', true ) @exclude.each { |x| announce( "\t #{x}", true ) } archive prepare replicate ensure cleanup end end private #---------------------------------------------------------------------# # Create, format and mount the blank machine image file. # TODO: investigate parallelizing prepare() with archive() def prepare FileUtils.mkdir_p( MOUNT ) announce( 'Creating and formatting file-system image...', true ) evaluate( "/usr/sbin/mkfile #{@size*1024*1024} #{@filename}" ) announce( 'Formatting file-system image...' ) execute( 'sync && devfsadm -C' ) @device = evaluate('/usr/sbin/lofiadm -a ' + @filename).strip number = @device.split(/\//).last rescue nil raise FatalError.new('Failed to attach image to a device' ) unless number execute( "echo y | newfs /dev/rlofi/#{number} < /dev/null > /dev/null 2>&1", true ) execute( 'sync' ) mount( @device, MOUNT ) end #---------------------------------------------------------------------# # Create a flash archive of the system at the desired volume root. def archive FileUtils.mkdir_p( WORKSPACE ) announce( 'Creating flash archive of file system...', true ) exempt = [] @exclude.each do |item| item = File.expand_path(item) # Since flarcreate does not allow you to exclude a mount-point from # a flash archive, we work around this by listing the files in that # directory and excluding them individually. if mounted? item exempt.concat( evaluate( 'ls -A ' + item).split(/\s/).map{|i| File.join( item, i ) } ) else exempt << item end end exempt = exempt.join( ' -x ') invocation = ['flar create -n ec2.archive -S -R ' + @volume ] invocation << ( '-x ' + exempt ) unless exempt.empty? invocation << ARCHIVE evaluate( invocation.join( ' ' ) ) raise FatalError.new( "Archive creation failed" ) unless File.exist?( ARCHIVE ) asize = FileUtil.size( ARCHIVE ) / ( 1024 * 1024 ) raise FatalError.new( "Archive too small" ) unless asize > 0 raise FatalError.new( 'Archive exceeds target size' ) if asize > @size end #---------------------------------------------------------------------# # Extract the archive into the file-system image. def replicate announce( 'Replicating archive to image (this will take a while)...', true ) # Extract flash archive into mounted image. The flar utility places # the output in a folder called 'archive'. Since we cannot override # this, we need to extract the content, move it to the image root # and delete remove the cruft extract = File.join( MOUNT, 'archive') execute( "flar split -S archive -d #{MOUNT} -f #{ARCHIVE}" ) execute( "ls -A #{extract} | xargs -i mv #{extract}/'{}' #{MOUNT}" ) FileUtils.rm_rf( File.join(MOUNT, 'archive') ) FileUtils.rm_rf( File.join(MOUNT, 'identification') ) announce 'Saving system configuration...' ['/boot/solaris/bootenv.rc', '/etc/vfstab', '/etc/path_to_inst'].each do |path| file = File.join( MOUNT, path ) FileUtils.cp( file, file + '.phys' ) end announce 'Fine-tuning system configuration...' execute( '/usr/sbin/sys-unconfig -R ' + MOUNT ) bootenv = File.join( MOUNT, '/boot/solaris/bootenv.rc' ) execute( "sed '/setprop bootpath/,/setprop console/d' < #{bootenv}.phys > #{bootenv}" ) execute( "sed '/dsk/d' < #{MOUNT}/etc/vfstab.phys > #{MOUNT}/etc/vfstab" ) FileUtils.rm_f( File.join(MOUNT, '/etc/rc2.d/S99dtlogin') ) announce 'Creating missing image directories...' [ '/dev/dsk', '/dev/rdsk', '/dev/fd', '/etc/mnttab', ].each do |item| FileUtils.mkdir_p( File.join( MOUNT, item ) ) end FileUtils.ln_s( '../../devices/xpvd/xdf@0:a', File.join( MOUNT, '/dev/dsk/c0d0s0' ) ) FileUtils.ln_s( '../../devices/xpvd/xdf@0:a,raw', File.join( MOUNT, '/dev/rdsk/c0d0s0' ) ) FileUtils.touch( File.join( MOUNT, Mtab::LOCATION ) ) fstab = File.join( MOUNT, Fstab::LOCATION ) File.open(fstab, 'w+') {|io| io << @vfstab } announce( "--->/etc/vfstab<---:\n" + @vfstab , true ) execute( "bootadm update-archive -R #{MOUNT} > /dev/null 2>&1", true ) announce( 'Disable xen services' ) file = File.join( MOUNT, '/var/svc/profile/upgrade' ) execute( 'echo "/usr/sbin/svcadm disable svc:/system/xctl/xend:default" >> ' + file ) announce 'Setting up DHCP boot' FileUtils.touch( File.join( MOUNT, '/etc/hostname.xnf0' ) ) FileUtils.touch( File.join( MOUNT, '/etc/dhcp.xnf0' ) ) announce 'Setting keyboard layout' kbd = File.join( MOUNT, '/etc/default/kbd' ) execute( "egrep '^LAYOUT' #{kbd} || echo 'LAYOUT=US-English' >> #{kbd}" ) end #---------------------------------------------------------------------# # Mount the specified device. The mount point is created if necessary. # We let mount guess the appropriate file system type. def mount(device, mpoint) FileUtils.mkdir_p(mpoint) if not FileUtil::exists?(mpoint) raise FatalError.new("image already mounted") if mounted?(mpoint) execute( 'sync' ) execute( 'mount ' + device + ' ' + mpoint ) end #---------------------------------------------------------------------# def unmount(mpoint, force=false) GC.start execute( 'sync && sync && sync' ) if mounted?( mpoint ) then execute( 'umount ' + (force ? '-f ' : '') + mpoint ) end end #---------------------------------------------------------------------# def mounted?(mpoint) EC2::Platform::Solaris::Mtab.load.entries.keys.include? mpoint end #---------------------------------------------------------------------# # Cleanup after self: # - unmount relevant mount points. # - release any device and resources attached to the image and mount-point # - delete any intermediate files and directories. def cleanup attempts = 0 begin unmount( MOUNT ) rescue ExecutionError announce "Unable to unmount image. Retrying after a short sleep." attempts += 1 if attempts < RETRIES sleep DELAY retry else announce( "Unable to unmount image after #{RETRIES} attempts. Baling out...", true ) unmount( MOUNT, true ) if File.exist?( @filename ) announce( "Deleting image file #{@filename}..." ) FileUtils.rm_f( @filename ) end end end unless @device.nil? devices = evaluate( 'lofiadm' ).split( /\n/ ) devices.each do |item| execute( 'lofiadm -d' + @device ) if item.index( @device ) == 0 end end execute( 'devfsadm -C' ) FileUtils.rm_rf( WORKSPACE ) if File.directory?( WORKSPACE ) end #---------------------------------------------------------------------# # Output a message if running in debug mode def announce(something, force=false) STDOUT.puts( something ) if @debug or force end #---------------------------------------------------------------------# # Execute the command line passed in. def execute( cmd, verbattim = false ) verbattim ||= @debug invocation = [ cmd ] invocation << ' 2>&1 > /dev/null' unless verbattim announce( "Executing: '#{cmd}' " ) time = Time.now raise ExecutionError.new( "Failed to execute '#{cmd}'.") unless system( invocation.join ) announce( "Time: #{Time.now - time}s", PROFILING ) end #---------------------------------------------------------------------# # Execute command line passed in and return STDOUT output if successful. def evaluate( cmd, success = 0, verbattim = false ) verbattim ||= @debug cmd << ' 2> /dev/null' unless verbattim announce( "Evaluating: '#{cmd}' " ) time = Time.now pid, stdin, stdout, stderr = Open4::popen4( cmd ) ignore stdin pid, status = Process::waitpid2 pid unless status.exitstatus == success raise ExecutionError.new( "Failed to evaluate '#{cmd }'. Reason: #{stderr.read}." ) end announce( "Time: #{Time.now - time}s", PROFILING ) stdout.read end def ignore(stuff) stuff end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/uname.rb0000644000000000000000000000143312050102225021073 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------- require 'ec2/platform/linux/uname' module EC2 module Platform module Solaris class Uname < EC2::Platform::Linux::Uname end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/architecture.rb0000644000000000000000000000216312050102225022451 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------- # Solaris machine architectures as seen in EC2 require 'ec2/platform/base/architecture' require 'ec2/platform/solaris/uname' module EC2 module Platform module Solaris class Architecture < EC2::Platform::Base::Architecture def self.bundling processor = Uname.processor return Architecture::I386 if processor =~ /^i\d86$/ return Architecture::X86_64 if processor =~ /^x86_64$/ return Architecture::UNKNOWN end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/solaris/rsync.rb0000644000000000000000000000151112050102225021121 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/fileutil' require 'ec2/platform/linux/rsync' module EC2 module Platform module Solaris class Rsync < EC2::Platform::Linux::Rsync EXECUTABLE = 'rsync' def initialize(e=EXECUTABLE) super e end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux.rb0000644000000000000000000000621512050102225017454 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ require 'ec2/platform/base' require 'ec2/platform/linux/identity' require 'ec2/platform/linux/architecture' require 'ec2/platform/linux/fstab' require 'ec2/platform/linux/mtab' require 'ec2/platform/linux/image' require 'ec2/platform/linux/rsync' require 'ec2/platform/linux/tar' require 'ec2/platform/linux/uname' require 'ec2/platform/linux/pipeline' require 'ec2/platform/linux/constants' module EC2 module Platform module Linux module Distribution include EC2::Platform::Base::Distribution REDHAT = 'Red Hat Linux' GENTOO = 'Gentoo' DEBIAN = 'Debian' UBUNTU = 'Ubuntu' FEDORA = 'Fedora' SLACKWARE = 'Slackware' SUSE = 'SuSE Linux' MANDRAKE = 'Mandrake' CAOS = 'Caos Linux' IDENTITIES= [ # file distro regex ['/etc/caos-release', Distribution::CAOS, nil], ['/etc/debian-release', Distribution::DEBIAN, nil], ['/etc/debian_version', Distribution::DEBIAN, nil], ['/etc/fedora-release', Distribution::FEDORA, nil], ['/etc/gentoo-release', Distribution::GENTOO, nil], ['/etc/redhat-release', Distribution::REDHAT, nil], ['/etc/slackware-version',Distribution::SLACKWARE, nil], ['/etc/slackware-release',Distribution::SLACKWARE, nil], ['/etc/SuSE-release', Distribution::SUSE, nil], ['/etc/ubuntu-release', Distribution::UBUNTU, nil], ['/etc/ubuntu-version', Distribution::UBUNTU, nil], ['/etc/mandrake-release', Distribution::MANDRAKE, nil], ] end class System < EC2::Platform::Base::System BUNDLING_ARCHITECTURE = EC2::Platform::Linux::Architecture.bundling #---------------------------------------------------------------------# def self.distribution Distribution::IDENTITIES.each do |file, distro, regex| if File.exists? file if regex.is_a? Regexp return distro if regex.match((IO.read file rescue nil)) else return distro end end end return Distribution::UNKNOWN end #---------------------------------------------------------------------# def self.superuser?() return `id -u`.strip == '0' end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/0000755000000000000000000000000012050102225017123 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/mtab.rb0000644000000000000000000000442112050102225020374 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. module EC2 module Platform module Linux LOCAL_FS_TYPES = ['ext2', 'ext3', 'xfs', 'jfs', 'reiserfs', 'tmpfs'] class Mtab class Entry REGEX = /^(\S+)\s+(\S+)\s+(\S+)\s+(\S+).*$/ attr_reader :device # mounted device. attr_reader :mpoint # mount point. attr_reader :fstype # file system type. attr_reader :options # options attr_reader :value # entire line def initialize(dev, mnt_point, fs_type, opts, line) @device = dev @mpoint = mnt_point @fstype = fs_type @options= opts @value = line end def self.parse(line) return nil if line[0,1] == '#' if (m = REGEX.match(line)) parts = m.captures return Entry.new(parts[0], parts[1], parts[2], parts[3], line.strip) else return nil end end def to_s value end def print puts(to_s) end end attr_reader :entries LOCATION = '/etc/mtab' def initialize(filename = LOCATION) begin f = File.new(filename, File::RDONLY) rescue SystemCallError => e raise FileError(filename, "could not open #{filename} to read mount table", e) end @entries = Hash.new f.readlines.each do |line| entry = Entry.parse(line) @entries[entry.mpoint] = entry unless entry.nil? end end def self.load self.new() end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/fstab.rb0000644000000000000000000000575412050102225020562 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ # An abstraction of the File-System Table (fstab) require 'ec2/amitools/version' module EC2 module Platform module Linux class Fstab include EC2Version class Entry REGEX = /^(\S+)\s+(\S+)\s+(\S+)\s+(\S+).*$/ attr_reader :device # mounted device. attr_reader :mpoint # mount point. attr_reader :fstype # file system type. attr_reader :options # options attr_reader :value # everything on line def initialize(dev, mnt_point, fs_type, opts, line) @device = dev @mpoint = mnt_point @fstype = fs_type @options= opts @value = line end def self.parse(line) return nil if line[0,1] == '#' if (m = REGEX.match(line)) parts = m.captures return Entry.new(parts[0], parts[1], parts[2], parts[3], line.strip) else return nil end end def to_s value end def print puts(to_s) end end LOCATION = '/etc/fstab' attr_reader :entries def initialize(filename = LOCATION) begin f = File.new(filename, File::RDONLY) rescue SystemCallError => e raise FileError(filename, "could not open #{filename} to read file system table", e) end @entries = Hash.new f.readlines.each do |line| entry = Entry.parse(line) @entries[entry.mpoint] = entry unless entry.nil? end end def self.load self.new() end DEFAULT = < #{file.path}" end command + list.join(' & ') end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/tar.rb0000644000000000000000000000775112050102225020250 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/platform/linux/constants' module EC2 module Platform module Linux class Tar class Command EXECUTABLE=EC2::Platform::Linux::Constants::Utility::TAR def initialize(e = EXECUTABLE) @files = [] @options = [] @executable = e end def version; @options << '--version'; self; end def verbose; @options << '-v'; self; end def create; @options << '-c'; self; end def bzip2; @options << '-j'; self; end def diff; @options << '-d'; self; end def gzip; @options << '-z'; self; end def extract; @options << '-x'; self; end def update; @options << '-u'; self; end def sparse; @options << '-S'; self; end def dereference; @options << '-h'; self; end def archive(filename) filename = '-' if filename.nil? @options << "-f #{filename}" self end def owner(user) @options << "--owner #{user}" self end def group(grp) @options << "--group #{grp}" self end def chdir(dir) @options << "-C #{dir}" unless dir.nil? self end def add(filename, dir = nil) item = dir.nil? ? filename : "-C #{dir} #{filename}" @files << item self end def expand "#{@executable} #{@options.join(' ')} #{@files.join(' ')}".strip end end class Version RECOMMENDED = 'tar 1.15' REGEX = /(?:tar).*?(\d+)\.(\d+)\.?(\d*)/ attr_reader :values attr_reader :string def initialize(str=nil) @string = str @string = default if str.nil? or str.empty? @values = Version.parse @string end def default s = `#{Command.new.version.expand}`.strip s = nil unless $? == 0 s end def string= (str) @string = str @values = Version.parse @string end def >= (other) return nil if @values.nil? if other.nil? or not other.is_a? Version raise ArgumentError, "Cannot compare with invalid version #{other}" end @values.zip(other.values).each do |mine, others| return false if mine < others return true if mine > others end return true end def usable? self >= Version.new(Version::RECOMMENDED) end def self.parse(str) match = REGEX.match(str) return nil if match.nil? begin items = match.captures.collect do |cap| cap.sub!(/^0*/, "") case cap when "" num = 0 else num = Integer(cap) end end rescue ArgumentError return nil end items end def self.current Version.new end end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/constants.rb0000644000000000000000000000163412050102225021470 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------ # Linux overrides for constants go here #------------------------------------------------------------------------ require 'ec2/platform/base/constants' module EC2 module Platform module Linux module Constants include EC2::Platform::Base::Constants end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/image.rb0000644000000000000000000002776712050102225020555 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'fileutils' require 'ec2/oem/open4' require 'ec2/amitools/fileutil' require 'ec2/amitools/syschecks' require 'ec2/amitools/exception' require 'ec2/platform/linux/mtab' require 'ec2/platform/linux/fstab' require 'ec2/platform/linux/constants' module EC2 module Platform module Linux # This class encapsulate functionality to create an file loopback image # from a volume. The image is created using dd. Sub-directories of the # volume, including mounts of local filesystems, are copied to the image. # Symbolic links are preserved. class Image IMG_MNT = '/mnt/img-mnt' EXCLUDES= ['/dev', '/media', '/mnt', '/proc', '/sys'] DEFAULT_FSTAB = EC2::Platform::Linux::Fstab::DEFAULT LEGACY_FSTAB = EC2::Platform::Linux::Fstab::LEGACY #---------------------------------------------------------------------# # Initialize the instance with the required parameters. # * _volume_ The path to the volume to create the image file from. # * _image_filename_ The name of the image file to create. # * _mb_image_size_ The image file size in MB. # * _exclude_ List of directories to exclude. # * _debug_ Make extra noise. def initialize( volume, image_filename, mb_image_size, exclude, includes, filter = true, fstab = nil, debug = false ) @volume = volume @image_filename = image_filename @mb_image_size = mb_image_size @exclude = exclude @includes = includes @filter = filter @fstab = nil # Cunning plan or horrible hack? # If :legacy is passed in as the fstab, we use the old v3 manifest's # device naming and fstab. if [:legacy, :default].include? fstab @fstab = fstab elsif not fstab.nil? @fstab = File.open(fstab).read() end @debug = debug # Exclude the temporary image mount point if it is under the volume # being bundled. if IMG_MNT.index( volume ) == 0 @exclude << IMG_MNT end end #--------------------------------------------------------------------# # Create the loopback image file and copy volume to it. def make begin puts( "Copying #{@volume} into the image file #{@image_filename}...") puts( 'Excluding: ' ) @exclude.each { |x| puts( "\t #{x}" ) } create_image_file format_image execute( 'sync' ) # Flush so newly formatted filesystem is ready to mount. mount_image make_special_dirs copy_rec( @volume, IMG_MNT) update_fstab ensure cleanup end end private #---------------------------------------------------------------------# def unmount(mpoint) if mounted?(mpoint) then execute('umount -d ' + mpoint) end end #---------------------------------------------------------------------# def mounted?(mpoint) EC2::Platform::Linux::Mtab.load.entries.keys.include? mpoint end #---------------------------------------------------------------------# # Unmount devices. Delete temporary files. def cleanup # Unmount image file. unmount(IMG_MNT) end #---------------------------------------------------------------------# # Call dd to create the image file. def create_image_file cmd = "dd if=/dev/zero of=" + @image_filename + " bs=1M count=1 seek=" + (@mb_image_size-1).to_s execute( cmd ) end #---------------------------------------------------------------------# # Format the image file, tune filesystem not to fsck based on interval. # Where available and possible, retain the original root volume label # uuid and file-system type falling back to using ext3 if not sure of # what to do. def format_image mtab = EC2::Platform::Linux::Mtab.load root = mtab.entries[@volume].device rescue nil info = fsinfo( root ) label= info[:label] uuid = info[:uuid] type = info[:type] || 'ext3' tune = nil mkfs = [ '/sbin/mkfs.' + type ] case type when 'btrfs' mkfs << [ '-L', label] if !label.to_s.empty? mkfs << [ @image_filename ] when 'xfs' mkfs << [ '-L', label] if !label.to_s.empty? mkfs << [ @image_filename ] tune = [ '/usr/sbin/xfs_admin' ] tune << [ '-U', uuid ] if uuid tune << [ @image_filename ] else # type unknown or ext2 or ext3 or ext4 mkfs << [ '-L', label] if !label.to_s.empty? mkfs << [ '-F', @image_filename ] tune = [ '/sbin/tune2fs -i 0' ] tune << [ '-U', uuid ] if uuid tune << [ @image_filename ] end execute( mkfs.join( ' ' ) ) execute( tune.join( ' ' ) ) if tune end def fsinfo( fs ) result = {} if fs and File.exists?( fs ) ['LABEL', 'UUID', 'TYPE' ].each do |tag| begin property = tag.downcase.to_sym value = evaluate( '/sbin/blkid -o value -s %s %s' % [tag, fs] ).strip result[property] = value if value and not value.empty? rescue FatalError => e if @debug STDERR.puts e.message STDERR.puts "Could not replicate file system #{property}. Proceeding..." end end end end result end #---------------------------------------------------------------------# # Mount the image file as a loopback device. The mount point is created # if necessary. def mount_image Dir.mkdir(IMG_MNT) if not FileUtil::exists?(IMG_MNT) raise FatalError.new("image already mounted") if mounted?(IMG_MNT) execute( 'mount -o loop ' + @image_filename + ' ' + IMG_MNT ) end #---------------------------------------------------------------------# # Copy the contents of the specified source directory to the specified # target directory, recursing sub-directories. Directories within the # exclusion list are not copied. Symlinks are retained but not traversed. # # src: The source directory name. # dst: The destination directory name. # options: A set of options to try. def copy_rec( src, dst, options={:xattributes => true} ) begin rsync = EC2::Platform::Linux::Rsync::Command.new rsync.archive.times.recursive.sparse.links.quietly.include(@includes).exclude(@exclude) if @filter rsync.exclude(EC2::Platform::Linux::Constants::Security::FILE_FILTER) end rsync.xattributes if options[ :xattributes ] rsync.src(File::join( src, '*' )).dst(dst) execute(rsync.expand) return true rescue Exception => e rc = $?.exitstatus return true if rc == 0 if rc == 23 and SysChecks::rsync_usable? STDERR.puts [ 'NOTE: rsync seemed successful but exited with error code 23. This probably means', 'that your version of rsync was built against a kernel with HAVE_LUTIMES defined,', 'although the current kernel was not built with this option enabled. The bundling', 'process will thus ignore the error and continue bundling. If bundling completes', 'successfully, your image should be perfectly usable. We, however, recommend that', 'you install a version of rsync that handles this situation more elegantly.' ].join("\n") return true elsif rc == 1 and options[ :xattributes ] STDERR.puts [ 'NOTE: rsync with preservation of extended file attributes failed. Retrying rsync', 'without attempting to preserve extended file attributes...' ].join("\n") o = options.clone o[ :xattributes ] = false return copy_rec( src, dst, o) end raise e end end #----------------------------------------------------------------------------# def make_special_dirs # Make /proc and /sys. Dir.mkdir( IMG_MNT + '/mnt' ) Dir.mkdir( IMG_MNT + '/proc' ) Dir.mkdir( IMG_MNT + '/sys' ) # Make device nodes. dev_dir = IMG_MNT + '/dev' Dir.mkdir( dev_dir ) # MAKEDEV is incredibly variable across distros, so use mknod directly. execute("mknod #{dev_dir}/null c 1 3") execute("mknod #{dev_dir}/zero c 1 5") execute("mknod #{dev_dir}/tty c 5 0") execute("mknod #{dev_dir}/console c 5 1") execute("ln -s null #{dev_dir}/X0R") end #----------------------------------------------------------------------------# def make_fstab case @fstab when :legacy return LEGACY_FSTAB when :default return DEFAULT_FSTAB else return @fstab end end #----------------------------------------------------------------------------# def update_fstab if @fstab etc = File::join( IMG_MNT, 'etc') fstab = File::join( etc, 'fstab' ) FileUtils::mkdir_p( etc ) unless File::exist?( etc) execute( "cp #{fstab} #{fstab}.old" ) if File.exist?( fstab ) fstab_content = make_fstab File.open( fstab, 'w' ) { |f| f.write( fstab_content ) } puts "/etc/fstab:" fstab_content.each do |s| puts "\t #{s}" end end end #----------------------------------------------------------------------------# # Execute the command line _cmd_. def execute( cmd ) if @debug STDERR.puts( "Executing: #{cmd} " ) suffix = '' else suffix = ' 2>&1 > /dev/null' end raise "execution failed: \"#{cmd}\"" unless system( cmd + suffix ) end #---------------------------------------------------------------------# # Execute command line passed in and return STDOUT output if successful. def evaluate( cmd, success = 0, verbattim = nil ) verbattim = @debug if verbattim.nil? STDERR.puts( "Evaluating: '#{cmd}' " ) if verbattim pid, stdin, stdout, stderr = Open4::popen4( cmd ) pid, status = Process::waitpid2 pid unless status.exitstatus == success raise FatalError.new( "Failed to evaluate '#{cmd }'. Reason: #{stderr.read}." ) end stdout.read end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/uname.rb0000644000000000000000000000265712050102225020567 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ostruct' require 'ec2/platform' module EC2 module Platform module Linux class Uname @@uname ||= OpenStruct.new def self.all @@uname.all ||= `uname -a`.strip end def self.platform @@uname.platform ||= `uname -i`.strip end def self.nodename @@uname.nodename ||= `uname -n`.strip end def self.processor @@uname.processor ||= `uname -p`.strip end def self.release @@uname.release ||= `uname -r`.strip end def self.os @@uname.os ||= `uname -s`.strip end def self.machine @@uname.machine ||= `uname -m`.strip end def self.uname @@uname end end end end end if __FILE__ == $0 include EC2::Platform::Linux puts "Uname = #{Uname.all.inspect}" end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/architecture.rb0000644000000000000000000000256512050102225022142 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ # Machine architectures as seen by EC2 in Linux require 'ec2/platform/base/architecture' require 'ec2/platform/linux/uname' module EC2 module Platform module Linux class Architecture < EC2::Platform::Base::Architecture #---------------------------------------------------------------------- # Returns the EC2-equivalent of the architecture of the platform this is # running on. def self.bundling processor = Uname.platform processor = Uname.machine if processor =~ /unknown/i return Architecture::I386 if processor =~ /^i\d86$/ return Architecture::X86_64 if processor =~ /^x86_64$/ return Architecture::UNKNOWN end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/linux/rsync.rb0000644000000000000000000001025112050102225020605 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/fileutil' module EC2 module Platform module Linux class Rsync class Command EXECUTABLE='rsync' def initialize(e = EXECUTABLE) @src = nil @dst = nil @options = [] @executable = e @quiet = false end def archive; @options << '-rlpgoD'; self; end def times; @options << '-t'; self; end def recursive; @options << '-r'; self; end def sparse; @options << '-S'; self; end def links; @options << '-l'; self; end def dereference; @options << '-L'; self; end def xattributes; @options << '-X'; self; end def version; @options << '--version'; self; end def src(path) @src = path; self; end def dst(path) @dst = path; self; end def quietly; @quiet = true; self; end alias :source :src alias :from :src alias :destination :dst alias :to :dst def exclude(files) if files.is_a? Array files.each {|file| exclude file } else @options << "--exclude #{files}" unless files.nil? end self end def include(files) if files.is_a? Array files.each {|file| include file } else @options << "--include #{files}" unless files.nil? end self end def expand "#{@executable} #{@options.join(' ')} #{@src} #{@dst} #{'2>&1 > /dev/null' if @quiet}".strip end end def self.symlinking? begin src = FileUtil.tempdir('ec2-ami-tools-rsync-test-src') dst = FileUtil.tempdir('ec2-ami-tools-rsync-test-dst') FileUtils.mkdir(src) FileUtils.touch("#{src}/foo") FileUtils.symlink("#{src}/foo", "#{src}/bar") FileUtils.mkdir("#{src}/baz") File.open("#{src}/baz/food", 'w+'){|io| io << IO.read(__FILE__) } FileUtils.symlink("#{src}/baz/food", "#{src}/baz/bard") FileUtils.mkdir(dst) incantation = Command.new.archive.recursive.sparse.links.src("#{src}/").dst("#{dst}") `#{incantation.expand} 2>&1` rc = $?.exitstatus return true if rc == 0 if rc == 23 #check that the structure was copied reasonably anyway slist = Dir["#{src}/**/**"] dlist = Dir["#{dst}/**/**"] return false unless dlist == dlist slist.each do |sitem| ditem = item.gsub(src, dst) return false unless dlist.include? ditem if File.file?(sitem) or File.symlink?(sitem) @out.print "comparing #{sitem} to #{ditem}" if @out return false unless IO.read(ditem) == IO.read(sitem) end if ['food', 'bard'].include? File.basename(ditem) return false unless IO.read(sitem) == IO.read(__FILE__) end end return true end return false rescue Exception return false ensure FileUtils.rm_rf src FileUtils.rm_rf dst end end def self.usable?() @@usable ||= self.symlinking? end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/base.rb0000644000000000000000000000240412050102225017223 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ module EC2 module Platform module Base module Distribution UNKNOWN = 'Unknown' GENERIC = 'Generic' end class System MOUNT_POINT = '/mnt/img-mnt' def self.distribution Distribution::UNKNOWN end def self.superuser? false end def self.exec(cmd, debug) if debug puts( "Executing: #{cmd} " ) suffix = '' else suffix = ' 2>&1 > /dev/null' end raise "execution failed: \"#{cmd}\"" unless system( cmd + suffix ) end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/base/0000755000000000000000000000000012050102225016676 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/platform/base/pipeline.rb0000644000000000000000000001471012050102225021033 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ require 'English' require 'fileutils' require 'tempfile' #------------------------------------------------------------------------------ module EC2 #---------------------------------------------------------------------------- module Platform #-------------------------------------------------------------------------- module Base #------------------------------------------------------------------------ class Pipeline #---------------------------------------------------------------------- class ExecutionError < RuntimeError def initialize(pipeline=nil, stage=nil, message=nil) word = 'Execution failed' word << ", pipeline: #{pipeline}" unless pipeline.nil? word << ", stage: #{stage}" unless stage.nil? word << ', mesage:' + message.to_s unless message.nil? word << '.' super word end end #---------------------------------------------------------------------- class Stage class Result attr :name attr :rc attr :successful def initialize(name, rc, successful) @name = name @rc = rc @successful = successful end def successful? @successful end def to_s "Result(name=#{@name}, rc=#{@rc}, successful=#{@successful})" end end attr :name attr :command attr :success def initialize(name, command, success=0) @name = name @command = command @success = success end def to_s() "Stage(name=#{@name}, command=#{@command}, success=#{@success})" end end attr_accessor :verbose attr_reader :basename #---------------------------------------------------------------------- def initialize(basename='pipeline', is_verbose=false) @stages = [] @results = [] @tempfiles = [] @basename = basename @verbose = is_verbose end #---------------------------------------------------------------------- def add(name, command, success=0) @stages << Stage.new(name, command, success) self end #---------------------------------------------------------------------- def concat(arr) if arr.is_a? Array arr.each do |e| self.add(e[0], e[1], e[2] || 0) end end self end #---------------------------------------------------------------------- # Given a pipeline of commands, modify it so that we can obtain # the exit status of each pipeline stage by reading the tempfile # associated with that stage. Must be implemented by subclasses def pipestatus(cmd) raise 'unimplemented method' end #---------------------------------------------------------------------- def command # Create the pipeline incantation pipeline = @stages.map { |s| s.command }.join(' | ') + '; ' # Fudge pipeline incantation to make return codes for each # stage accessible from the associated pipeline stage pipestatus(pipeline) end #---------------------------------------------------------------------- def execute() @results = [] create_tempfiles escaped_command = command.gsub("'","'\"'\"'") invocation = "/bin/bash -c '#{escaped_command}'" # Execute the pipeline invocation STDERR.puts("Pipeline.execute: command = [#{invocation}]") if verbose output = `#{invocation}` STDERR.puts("Pipeline.execute: output = [#{output.strip}]") if verbose unless $CHILD_STATUS.success? raise ExecutionError.new(@basename) end # Collect the pipeline's exit codes and see if they're good successful = true offender = nil @results = @tempfiles.zip(@stages).map do |file, stage| file.open() status = file.read().strip.to_i file.close(false) success = (stage.success == status) successful &&= success offender = stage.name unless successful Stage::Result.new(stage.name, status, success) end unless successful raise ExecutionError.new(@basename, offender) end output end #---------------------------------------------------------------------- def cleanup @tempfiles.each do |file| file.close(true) if file.is_a? Tempfile FileUtils.rm_f(file.path) if File.exist?(file.path) end end #---------------------------------------------------------------------- def errors @results.reject { |r| r.success } end #---------------------------------------------------------------------- def create_tempfiles @tempfiles = (0...@stages.length).map do |index| file = Tempfile.new("#{@basename}-pipestatus-#{index}") file.close(false) file end unless @tempfiles.length == @stages.length raise ExecutionError.new( @basename, nil, "Temp files count(#{@tempfiles.length}) != stages count(#{@stages.length})") end end #---------------------------------------------------------------------- def to_s "Pipeline(stages=[#{@stages.join(', ')}], results=[#{@results.join(', ')}])" end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/base/constants.rb0000644000000000000000000000361612050102225021245 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #-------------------------------------------------------------------------- # Definition of constant values used by the AMI tools #------------------------------------------------------------------------ module EC2 module Platform module Base module Constants module Bundling EC2_HOME = ENV["EC2_AMITOOL_HOME"] || ENV["EC2_HOME"] EC2_X509_CERT = File.join(EC2_HOME.to_s, '/etc/ec2/amitools/cert-ec2.pem') EC2_X509_GOV_CERT = File.join(EC2_HOME.to_s, '/etc/ec2/amitools/cert-ec2-gov.pem') EC2_MAPPING_FILE = File.join(EC2_HOME.to_s, '/etc/ec2/amitools/mappings.csv') EC2_MAPPING_URL = 'https://ec2-downloads.s3.amazonaws.com/mappings.csv' DESTINATION = '/tmp' end module Utility OPENSSL = 'openssl' RSYNC = 'rsync' TAR = 'tar' TEE = 'tee' GZIP = 'gzip' end module Security FILE_FILTER = [ '"*/#*#"', '"*/.#*"', '"*.sw"', '"*.swo"', '"*.swp"', '"*~"', '"*.pem"', '"*.priv"', '"*id_rsa*"', '"*id_dsa*"', '"*.gpg"', '"*.jks"', '"*/.ssh/authorized_keys"', '"*/.bash_history"'] end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/base/architecture.rb0000644000000000000000000000153112050102225021705 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. module EC2 module Platform module Base class Architecture I386 = 'i386' X86_64 = 'x86_64' UNKNOWN = 'unknown' SUPPORTED = [I386, X86_64] def self.supported? arch SUPPORTED.include? arch end end end end end ec2-ami-tools-1.4.0.9/lib/ec2/platform/current.rb0000644000000000000000000000360212050102225017774 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. #------------------------------------------------------------------------------ require 'pathname' require 'ec2/platform' module EC2 module Platform class Unknown < RuntimeError def initialize(name) super("Unknown platform: #{name}") @name = name end attr_reader :name end class Unsupported < RuntimeError def initialize(name) super("Unsupported or unimplemented platform: #{name}") @name = name end attr_reader :name end def self.initialize return EC2::Platform::PEER if defined? EC2::Platform::PEER impl = Platform::IMPL base = impl.to_s # must be a known architecture raise Unknown.new(base), caller if base.nil? or impl == :unknown # base file must exist in same directory as this one file = Pathname.new(__FILE__).dirname + base raise Unsupported.new(base), caller unless File.exists? file # a require statement must succeed implemented = require "ec2/platform/#{base}" rescue false raise Unsupported.new(impl), caller unless implemented # cross fingers and hope the 'required' peer set the PEER constant raise Unsupported.new(impl), caller unless defined? EC2::Platform::PEER EC2::Platform::PEER end Current = initialize end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/0000755000000000000000000000000012050102225015767 5ustar rootrootec2-ami-tools-1.4.0.9/lib/ec2/amitools/decryptmanifest.rb0000644000000000000000000000133112050102225021513 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/crypto' USAGE = "Usage: decryptmanifest private-key-path" if ARGV.size < 1 puts USAGE exit 1 end puts Crypto::decryptasym( $stdin.read, ARGV[0] ) ec2-ami-tools-1.4.0.9/lib/ec2/amitools/syschecks.rb0000644000000000000000000000172112050102225020314 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/fileutil' require 'ec2/platform/current' module SysChecks def self.rsync_usable?() EC2::Platform::Current::Rsync.usable? end def self.good_tar_version?() EC2::Platform::Current::Tar::Version.current.usable? end def self.get_system_arch() EC2::Platform::Current::System::BUNDLING_ARCHITECTURE end def self.root_user?() EC2::Platform::Current::System.superuser? end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/exception.rb0000644000000000000000000000503612050102225020316 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. ## # An exception thrown to indicate an unrecoverable error has been encountered. # The process should be terminated and the exception's message should be # displayed to the user. # class FatalError < Exception ## # ((|message|)) The message that should be displayed to the user. # ((|cause|)) The exception that caused the fatal error. # def initialize(message, cause = nil) super(message) @cause = cause end end #------------------------------------------------------------------------------# ## # File access error. # class FileError < FatalError def initialize(filename, error_description, sys_call_err = nil) message = "File Error: #{error_description} \n" + "File name: #{filename}\n" super(message, sys_call_err) end end #------------------------------------------------------------------------------# ## # Directory access exception. # class DirectoryError < FatalError def initialize(dirname, error_description, sys_call_err = nil) message = "Directory Error: #{error_description} \n" + "Directory name: #{dirname} \n" super(message, sys_call_error) end end #----------------------------------------------------------------------------# class DownloadError < FatalError def initialize(resource, addr, port, path, error=nil) super("could not download #{resource} from #{addr}/#{path} on #{port}", error) end end #----------------------------------------------------------------------------# class UploadError < FatalError def initialize(resource, addr, port, path, error=nil) super("could not upload #{resource} to #{addr}/#{path} on #{port}", error) end end #------------------------------------------------------------------------------# ## # Parameter error. # class ParameterError < FatalError def initialize(message) super(message) end end #------------------------------------------------------------------------------# class AMIInvalid < FatalError def initialize(message) super(message) end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/manifest_wrapper.rb0000644000000000000000000001005312050102225021661 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/manifestv20071010' require 'ec2/amitools/manifestv20070829' require 'ec2/amitools/manifestv3' require 'rexml/document' class ManifestWrapper class InvalidManifest < RuntimeError end # All the manifest fields we support. V3_FIELDS = [ :name, :user, :parts, :size, :bundled_size, :user_encrypted_key, :ec2_encrypted_key, :cipher_algorithm, :user_encrypted_iv, :ec2_encrypted_iv, :digest, :digest_algorithm, :bundler_name, :bundler_version, :bundler_release, ] V3_FIELDS.each { |field| attr_reader(field) } V20070829_FIELDS = [ :arch, ] V20070829_FIELDS.each { |field| attr_reader(field) } V20071010_FIELDS = [ :image_type, :kernel_id, :ramdisk_id, :product_codes, :ancestor_ami_ids, :block_device_mapping, :kernel_name, ] V20071010_FIELDS.each { |field| attr_reader(field) } # We want to pass some methods through as well. METHODS = [ :authenticate, ] METHODS.each do |method| define_method(method) do |*args| @manifest.send(method, *args) end end # Should the caller want the underlying manifest for some reason. attr_reader :manifest def get_manifest_version(manifest_xml) begin version_elem = REXML::XPath.first(REXML::Document.new(manifest_xml), '/manifest/version') raise InvalidManifest.new("Invalid manifest.") if version_elem.nil? return version_elem.text.to_i rescue => e raise InvalidManifest.new("Invalid manifest.") end end def initialize(manifest_xml) version = get_manifest_version(manifest_xml) if version > 20071010 raise InvalidManifest.new("Manifest is too new for this tool to handle. Please upgrade.") end if version < 3 raise InvalidManifest.new("Manifest is too old for this tool to handle.") end # Try figure out what manifest version we have @manifest = if ManifestV20071010::version20071010?(manifest_xml) ManifestV20071010.new(manifest_xml) elsif ManifestV20070829::version20070829?(manifest_xml) ManifestV20070829.new(manifest_xml) elsif ManifestV3::version3?(manifest_xml) ManifestV3.new(manifest_xml) else raise InvalidManifest.new("Unrecognised manifest version.") end # Now populate the fields. First, stuff that's in all the # manifests we deal with. V3_FIELDS.each do |field| instance_variable_set("@#{field.to_s}", @manifest.send(field)) end # Next, the next version up. if @manifest.version > 3 V20070829_FIELDS.each do |field| instance_variable_set("@#{field.to_s}", @manifest.send(field)) end else # Some mandatory fields we need in later versions: @arch = 'i386' end # Next, the next version up. if @manifest.version > 20070829 V20071010_FIELDS.each do |field| instance_variable_set("@#{field.to_s}", @manifest.send(field)) end else # Some mandatory fields we need in later versions: @image_type = 'machine' end end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/instance-data.rb0000644000000000000000000000464512050102225021040 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'open-uri' module EC2 class InstanceData META_DATA_URL = "http://169.254.169.254/latest/meta-data/" attr_reader :instance_data_accessible def initialize(meta_data_url = META_DATA_URL) @meta_data_url = meta_data_url # see if we can access the meta data. Be unforgiving - if anything goes wrong # just mark instance data as unaccessible. begin open(@meta_data_url) @instance_data_accessible = true rescue StandardError => e @instance_data_accessible = false end end def kernel_id read_meta_data('kernel-id') end def ramdisk_id read_meta_data('ramdisk-id') end def ami_id read_meta_data('ami-id') end def ancestor_ami_ids read_meta_data_list('ancestor-ami-ids') end def product_codes read_meta_data_list('product-codes') end def block_device_mapping read_meta_data_hash('block-device-mapping') end def availability_zone read_meta_data('placement/availability-zone') end def read_meta_data_hash(path) keys = list_meta_data_index(path) return nil if keys.nil? hash = {} keys.each do |key| value = read_meta_data(File.join(path, key)) hash[key] = value if value end hash end private :read_meta_data_hash def read_meta_data_list(path) list = read_meta_data(path) list.nil? ? nil : list.split("\n") end private :read_meta_data_list def list_meta_data_index(path) read_meta_data_list(File.join(path, '')) end private :list_meta_data_index def read_meta_data(path) nil if !@instance_data_accessible begin open(File.join(@meta_data_url, path)) do |cio| return cio.read.to_s.strip end rescue return nil end end private :read_meta_data end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/downloadbundleparameters.rb0000644000000000000000000000565012050102225023407 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/s3toolparameters' #------------------------------------------------------------------------------# class DownloadBundleParameters < S3ToolParameters PREFIX_DESCRIPTION = "The filename prefix for bundled AMI files. Defaults to 'image'." DIRECTORY_DESCRIPTION = ['The directory into which to download the bundled AMI parts.', "Defaults to the current working directory."] MANIFEST_DESCRIPTION = ["The local manifest filename. Required only for manifests that", "pre-date the version 3 manifest file format."] RETRY_DESCRIPTION = "Automatically retry failed downloads." attr_accessor :manifest, :prefix, :privatekey, :directory, :retry #----------------------------------------------------------------------------# def mandatory_params() super() on('-k', '--privatekey KEY', String, USER_PK_PATH_DESCRIPTION) do |privatekey| assert_file_exists(privatekey, '--privatekey') @privatekey = privatekey end end #----------------------------------------------------------------------------# def optional_params() super() on('-m', '--manifest FILE', String, *MANIFEST_DESCRIPTION) do |manifest| assert_good_key(manifest, '--manifest') @manifest = manifest end on('-p', '--prefix PREFIX', String, PREFIX_DESCRIPTION) do |prefix| assert_good_key(prefix, '--prefix') @prefix = prefix end on('-d', '--directory DIRECTORY', String, *DIRECTORY_DESCRIPTION) do |directory| assert_directory_exists(directory, '--directory') @directory = directory end on('--retry', RETRY_DESCRIPTION) do @retry = true end end #----------------------------------------------------------------------------# def validate_params() super() raise MissingMandatory.new('--privatekey') unless @privatekey raise InvalidCombination.new('--prefix', '--manifest') if (@prefix and @manifest) end #----------------------------------------------------------------------------# def set_defaults() super() @directory = Dir::pwd() unless @directory @prefix = @manifest.split('.')[0..-2].join('.') if (@manifest) @prefix = 'image' unless @prefix @manifest = "#{@prefix}.manifest.xml" unless @manifest end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/migratebundleparameters.rb0000644000000000000000000001446112050102225023230 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/parameters_base' require 'ec2/platform/current' require 'ec2/amitools/region' #------------------------------------------------------------------------------# class MigrateBundleParameters < ParametersBase include EC2::Platform::Current::Constants MANIFEST_DESCRIPTION = "The name the manifest file." DIRECTORY_DESCRIPTION = ["The directory containing the bundled AMI parts to upload.", "Defaults to the directory containing the manifest."] USER_CERT_PATH_DESCRIPTION = "The path to the user's PEM encoded RSA public key certificate file." USER_PK_PATH_DESCRIPTION = "The path to the user's PEM encoded RSA private key file." EC2_CERT_PATH_DESCRIPTION = ['The path to the EC2 X509 public key certificate bundled into the AMI.', "Defaults to '#{Bundling::EC2_X509_CERT}'."] KERNEL_DESCRIPTION = "Kernel id to bundle into the AMI." RAMDISK_DESCRIPTION = "Ramdisk id to bundle into the AMI." DEST_BUCKET_DESCRIPTION = "The bucket to copy bundle to. Created if nonexistent." BUCKET_DESCRIPTION = "The bucket containing the AMI to be migrated." USER_DESCRIPTION = "The user's AWS access key ID." PASS_DESCRIPTION = "The user's AWS secret access key." ACL_DESCRIPTION = ["The access control list policy [\"public-read\" | \"aws-exec-read\"].", "Defaults to \"aws-exec-read\"."] URL_DESCRIPTION = "The S3 service URL. Defaults to https://s3.amazonaws.com." RETRY_DESCRIPTION = "Automatically retry failed uploads. Use with caution." LOCATION_DESCRIPTION = "The location of the bucket to upload to [#{AwsRegion.regions.join(',')}]." NO_MAPPING_DESCRIPTION = "Do not perform automatic mappings." REGION_DESCRIPTION = "Region to look up in the mapping file." attr_accessor :user_pk_path, :user_cert_path, :ec2_cert_path, :user, :pass, :kernel_id, :ramdisk_id, :s3_url, :bucket, :keyprefix, :dest_bucket, :dest_keyprefix, :manifest_name, :location, :acl, :retry, :use_mapping, :region def split_container(container) splitbits = container.sub(%r{^/*},'').sub(%r{/*$},'').split("/") bucket = splitbits.shift keyprefix = splitbits.join("/") keyprefix += "/" unless keyprefix.empty? [bucket, keyprefix] end def mandatory_params() on('-c', '--cert PATH', String, USER_CERT_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--cert') @user_cert_path = path end on('-k', '--privatekey PATH', String, USER_PK_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--privatekey') @user_pk_path = path end on('-m', '--manifest NAME', String, MANIFEST_DESCRIPTION) do |manifest| raise InvalidValue.new("--manifest", manifest) unless manifest =~ /\.manifest\.xml$/ assert_good_key(manifest, '--manifest') @manifest_name = manifest end on('-b', '--bucket BUCKET', String, BUCKET_DESCRIPTION) do |container| @container = container @bucket, @keyprefix = split_container(@container) end on('-d', '--destination-bucket BUCKET', String, DEST_BUCKET_DESCRIPTION) do |dest_container| @dest_container = dest_container @dest_bucket, @dest_keyprefix = split_container(@dest_container) end on('-a', '--access-key USER', String, USER_DESCRIPTION) do |user| @user = user end on('-s', '--secret-key PASSWORD', String, PASS_DESCRIPTION) do |pass| @pass = pass end end def optional_params() on('--ec2cert PATH', String, *EC2_CERT_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--ec2cert') @ec2_cert_path = path end on('--acl ACL', String, *ACL_DESCRIPTION) do |acl| raise InvalidValue.new('--acl', acl) unless ['public-read', 'aws-exec-read'].include?(acl) @acl = acl end on('--url URL', String, URL_DESCRIPTION) do |url| @s3_url = url end on('--retry', RETRY_DESCRIPTION) do @retry = true end on('--location LOCATION', LOCATION_DESCRIPTION) do |location| assert_option_in(location, AwsRegion.s3_locations, '--location') @location = location @location = :unconstrained if @location == "US" end on('--kernel KERNEL_ID', String, KERNEL_DESCRIPTION) do |kernel_id| @kernel_id = kernel_id end on('--ramdisk RAMDISK_ID', String, RAMDISK_DESCRIPTION) do |ramdisk_id| @ramdisk_id = ramdisk_id end on('--no-mapping', String, NO_MAPPING_DESCRIPTION) do @use_mapping = false end on('--region REGION', String, REGION_DESCRIPTION) do |region| @region = region end end def validate_params() raise MissingMandatory.new('--manifest') unless @manifest_name raise MissingMandatory.new('--cert') unless @user_cert_path raise MissingMandatory.new('--privatekey') unless @user_pk_path raise MissingMandatory.new('--bucket') unless @container raise MissingMandatory.new('--destination-bucket') unless @dest_container raise MissingMandatory.new('--access-key') unless @user raise MissingMandatory.new('--secret-key') unless @pass end def set_defaults() @acl ||= 'aws-exec-read' @s3_url ||= 'https://s3.amazonaws.com' @ec2_cert_path ||= Bundling::EC2_X509_CERT @use_mapping = true if @use_mapping.nil? # False is different. end def help() s = "*** This tool is deprecated. ***\n" s += "Please consider using ec2-migrate-image from the EC2 API tools.\n" s += "You can download the EC2 API tools here:\n" s += "http://aws.amazon.com/developertools/351\n" s += super() end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundleimage.rb0000644000000000000000000000564012050102225020575 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/bundle' require 'ec2/amitools/bundleimageparameters' require 'ec2/amitools/bundle_base' MAX_SIZE = 10 * 1024 * 1024 * 1024 # 10 GB in bytes. BUNDLE_IMAGE_NAME = 'ec2-bundle-image' # The manual. BUNDLE_IMAGE_MANUAL=< MAX_SIZE raise "the specified image #{p.image_path} is too large" end else $stderr.puts 'Warning: disabling size-checks can result in unbootable image' end optional_args = { :kernel_id => p.kernel_id, :ramdisk_id => p.ramdisk_id, :product_codes => p.product_codes, :ancestor_ami_ids => p.ancestor_ami_ids, :block_device_mapping => p.block_device_mapping, } $stdout.puts 'Bundling image file...' Bundle.bundle_image(File::expand_path(p.image_path), p.user, p.arch, Bundle::ImageType::MACHINE, p.destination, p.user_pk_path, p.user_cert_path, p.ec2_cert_path, p.prefix, optional_args, @debug, false) $stdout.puts( "#{BUNDLE_IMAGE_NAME} complete." ) end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() BUNDLE_IMAGE_MANUAL end def get_name() BUNDLE_IMAGE_NAME end def main(p) bundle_image(p) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 ImageBundler.new().run(BundleImageParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/manifestv3.rb0000644000000000000000000002456412050102225020406 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/crypto' require 'ec2/amitools/format' require 'ec2/amitools/xmlutil' require 'pathname' require 'rexml/document' # Manifest Version 3. # Not backwards compatible class ManifestV3 VERSION_STRING = '3' VERSION = VERSION_STRING.to_i # AMI part information container. class PartInformation attr_reader :filename # Part's file basename. attr_reader :digest # Part's digest hex encoded. # Initialize with part's _filename_ and _digest_ as byte string. def initialize( filename, digest ) @filename , @digest = filename, digest end end def initialize( xml = nil ) if xml == nil @doc = REXML::Document.new else # Convert to string if necessary. xml = ( xml.kind_of?( IO ) ? xml.read : xml ) @doc = REXML::Document.new( xml ) end end # for debugging only def doc @doc end # Initialize the manifest with AMI information. # Return +true+ if the initialization was succesful. # Raise an exception on error. def init( name, user, # The user's account number. parts, # A list of parts filenames and digest pairs. size, # The size of the AMI in bytes. bundled_size, # The size of the bunled AMI in bytes. user_encrypted_key, # Hex encoded. ec2_encrypted_key, # Hex encoded. cipher_algorithm, # The cipher algorithm used to encrypted the AMI. user_encrypted_iv, # Hex encoded. ec2_encrypted_iv, # Hex encoded. digest, # Hex encoded. digest_algorithm, # The digest algorithm. privkey_filename, # The user's private key filename. bundler_name = nil, bundler_version = nil, bundler_release = nil ) # Check non-String parameter types. raise ArgumentError.new( "parts parameter type invalid" ) unless parts.is_a? Array # XML document. @doc = REXML::Document.new @doc << REXML::XMLDecl.new # manifest - the root element. manifest = REXML::Element.new( 'manifest' ) @doc.add_element( manifest ) # version - indicate the manifest version. version = REXML::Element.new( 'version' ) version.text = VERSION_STRING manifest.add_element( version ) # bundler information if bundler_name or bundler_version or bundler_release bundler_element = REXML::Element.new( 'bundler' ) manifest.add_element( bundler_element ) [['name', bundler_name ], ['version', bundler_version ], ['release', bundler_release ]].each do |element_name, text| if element_name element = REXML::Element.new( element_name ) element.text = text bundler_element.add_element( element ) end end end # image - the image element. image = REXML::Element.new( 'image' ) name_element = REXML::Element.new( 'name' ) name_element.text = name image.add_element( name_element ) manifest.add_element( image ) # user - the user's AWS access key ID. user_element = REXML::Element.new( 'user' ) user_element.text = user image.add_element( user_element ) # digest - the digest of the AMI. digest_element = REXML::Element.new( 'digest' ) digest_element.add_attribute( 'algorithm', digest_algorithm ) digest_element.add_text( digest ) image.add_element( digest_element ) # size - the size of the uncompressed AMI. size_element = REXML::Element.new( 'size' ) size_element.text = size.to_s image.add_element( size_element ) # size - the size of the uncompressed AMI. bundled_size_element = REXML::Element.new( 'bundled_size' ) bundled_size_element.text = bundled_size.to_s image.add_element( bundled_size_element ) # ec2 encrypted key element. ec2_encrypted_key_element = REXML::Element.new( 'ec2_encrypted_key' ) ec2_encrypted_key_element.add_attribute( 'algorithm', cipher_algorithm ) ec2_encrypted_key_element.add_text( ec2_encrypted_key ) image.add_element( ec2_encrypted_key_element ) # user encrypted key element. user_encrypted_key_element = REXML::Element.new( 'user_encrypted_key' ) user_encrypted_key_element.add_attribute( 'algorithm', cipher_algorithm ) user_encrypted_key_element.add_text( user_encrypted_key ) image.add_element( user_encrypted_key_element ) # ec2 encrypted iv element. ec2_encrypted_iv_element = REXML::Element.new( 'ec2_encrypted_iv' ) ec2_encrypted_iv_element.add_text( ec2_encrypted_iv ) image.add_element( ec2_encrypted_iv_element ) # user encrypted iv element. user_encrypted_iv_element = REXML::Element.new( 'user_encrypted_iv' ) user_encrypted_iv_element.add_text( user_encrypted_iv ) image.add_element( user_encrypted_iv_element ) # parts - list of the image parts. parts_element = REXML::Element.new( 'parts' ) parts_element.add_attributes( {'count' => parts.size.to_s} ) index=0 parts.each do |part| # Add image part element for each image part. part_element = REXML::Element.new( 'part' ) part_element.add_attribute( 'index', index.to_s ) filename = REXML::Element.new( 'filename' ) filename.add_text( part[0] ) part_element.add_element( filename ) digest = REXML::Element.new( 'digest' ) digest.add_attribute( 'algorithm', digest_algorithm ) digest.add_text( Format::bin2hex( part[1] ) ) part_element.add_element( digest ) parts_element.add_element( part_element ) index+=1 end image.add_element( parts_element ) # Sign the manifest. sign( privkey_filename ) return true end def ManifestV3::version3?( xml ) doc = REXML::Document.new( xml ) version = REXML::XPath.first( doc.root, 'version' ) return (version and version.text and version.text.to_i == VERSION) end # Return the AMI's digest hex encoded. def digest() return get_element_text( 'image/digest' ) end def name() return get_element_text( 'image/name' ) end # The ec2 encrypted key hex encoded. def ec2_encrypted_key() return get_element_text('image/ec2_encrypted_key' ) end # The user encrypted key hex encoded. def user_encrypted_key() return get_element_text( 'image/user_encrypted_key' ) end # The ec2 encrypted initialization vector hex encoded. def ec2_encrypted_iv() return get_element_text( 'image/ec2_encrypted_iv' ) end # The user encrypted initialization vector hex encoded. def user_encrypted_iv() return get_element_text( 'image/user_encrypted_iv' ) end # Get digest algorithm used. def digest_algorithm() return REXML::XPath.first(@doc.root, 'image/digest/@algorithm').to_s end # Get cipher algorithm used. def cipher_algorithm() return REXML::XPath.first(@doc.root, 'image/ec2_encrypted_key/@algorithm').to_s end # Retrieve a list of AMI bundle parts info. Each element is a hash # with the following elements: # * 'digest' # * 'filename' # * 'index' def ami_part_info_list parts = Array.new REXML::XPath.each( @doc.root,'image/parts/part' ) do |part| index = part.attribute( 'index' ).to_s.to_i filename = REXML::XPath.first( part, 'filename' ).text digest = REXML::XPath.first( part, 'digest' ).text parts << { 'digest'=>digest, 'filename'=>filename, 'index'=>index } end return parts end # A list of PartInformation instances representing the AMI parts. def parts() parts = [] REXML::XPath.each( @doc.root,'image/parts/part' ) do |part| index = part.attribute( 'index' ).to_s.to_i filename = REXML::XPath.first( part, 'filename' ).text digest = Format::hex2bin( REXML::XPath.first( part, 'digest' ).text ) parts[index] = PartInformation.new( filename, digest ) end return parts end # Return the size of the AMI. def size() return get_element_text( 'image/size' ).to_i() end # Return the bundled size of the AMI. def bundled_size() return get_element_text( 'image/bundled_size' ).to_i end # Return the bundler name. def bundler_name() return get_element_text('bundler/name') end # Return the bundler version. def bundler_version() return get_element_text('bundler/version') end # Return the bundler release. def bundler_release() return get_element_text('bundler/release') end # Sign the manifest. If it is already signed, the signature and certificate # will be replaced def sign( privkey_filename ) unless privkey_filename.kind_of? String and File::exist?( privkey_filename ) raise ArgumentError.new( "privkey_filename parameter invalid" ) end # Get the XML for image element and sign it. image_xml = XMLUtil.get_xml( @doc.to_s, 'image' ) sig = Crypto::sign( image_xml, privkey_filename ) # Create the signature and certificate elements. signature = REXML::Element.new( 'signature' ) signature.add_text( Format::bin2hex( sig ) ) @doc.root.delete_element( 'signature' ) @doc.root.add_element( signature ) end # Return the signature def signature get_element_text('signature') end # Verify the signature def authenticate(cert) image_xml = XMLUtil.get_xml( @doc.to_s, 'image' ) pubkey = Crypto::cert2pubkey(cert) Crypto::authenticate(image_xml, Format::hex2bin(signature), pubkey) end # Return the manifest as an XML string. def to_s() return @doc.to_s end def user() return get_element_text( 'image/user' ) end def version() return get_element_text( 'version' ).to_i end private def get_element_text( xpath ) element = REXML::XPath.first( @doc.root, xpath ) unless element raise "invalid AMI manifest, #{xpath} element not present" end unless element.text raise "invalid AMI manifest, #{xpath} element empty" end return element.text end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/migratebundle.rb0000644000000000000000000001773412050102225021152 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/common/s3support' require 'ec2/amitools/region' require 'ec2/amitools/migratebundleparameters' require 'ec2/amitools/migratemanifest' require 'ec2/amitools/downloadbundle' require 'ec2/amitools/uploadbundle' require 'fileutils' require 'ec2/amitools/tool_base' MIGRATE_BUNDLE_NAME = 'ec2-migrate-bundle' #------------------------------------------------------------------------------# MIGRATE_BUNDLE_MANUAL =< e # Nuke it FileUtils::rm_rf(tempdir) raise e end # Nuke it FileUtils::rm_rf(tempdir) result end #------------------------------------------------------------------------------# def uri2string(uri) s = "#{uri.scheme}://#{uri.host}:#{uri.port}#{uri.path}" # Remove the trailing '/'. return (s[-1..-1] == "/" ? s[0..-2] : s) end #------------------------------------------------------------------------------# def make_s3_connection(s3_url, user, pass, bucket) s3_uri = URI.parse(s3_url) s3_url = uri2string(s3_uri) v2_bucket = EC2::Common::S3Support::bucket_name_s3_v2_safe?(bucket) EC2::Common::S3Support.new(s3_url, user, pass, (v2_bucket ? nil : :path)) end #------------------------------------------------------------------------------# def download_manifest(s3_conn, bucket, manifest_name, manifest_path, user_pk_path, retry_stuff) BundleDownloader.new().download_manifest(s3_conn, bucket, manifest_name, manifest_path, user_pk_path, retry_stuff) end #------------------------------------------------------------------------------# def get_part_filenames(manifest_path, user_cert_path) manifest = ManifestMigrator.new().get_manifest(manifest_path, user_cert_path) manifest.parts.collect { |part| part.filename }.sort end #------------------------------------------------------------------------------# def copy_part(s3_conn, bucket, keyprefix, dest_bucket, dest_keyprefix, part, acl, retry_copy) source = "/#{bucket}/#{keyprefix}#{part}" retry_s3(retry_copy) do begin s3_conn.copy(dest_bucket, dest_keyprefix+part, source, {"x-amz-acl"=>acl}) return rescue => e raise TryFailed.new("Failed to copy \"#{part}\": #{e.message}") end end end #------------------------------------------------------------------------------# def upload_manifest(s3_conn, key, manifest_path, bucket, tempdir, acl, retry_stuff) BundleUploader.new().upload(s3_conn, bucket, key, manifest_path, acl, retry_stuff) end #------------------------------------------------------------------------------# def migrate_bundle(s3_url, bucket, keyprefix, dest_bucket, dest_keyprefix, manifest_name, user_pk_path, user_cert_path, user, pass, location, kernel_id=nil, ramdisk_id=nil, acl='aws-exec-read', retry_stuff=nil, use_mapping=true, region=nil) src_s3_conn = make_s3_connection(s3_url, user, pass, bucket) dest_s3_conn = make_s3_connection(s3_url, user, pass, dest_bucket) # Locate destination bucket and create it if necessary. bu = BundleUploader.new() bu.check_bucket_name(dest_bucket) bucket_location = bu.get_bucket_location(dest_s3_conn, dest_bucket) bu.create_bucket(dest_s3_conn, dest_bucket, bucket_location, location, retry_stuff) # Region/location hack: if region.nil? location ||= bucket_location region = AwsRegion.guess_region_from_s3_bucket(location) puts "Region not provided, guessing from S3 location: #{region}" end with_temp_dir(manifest_name) do |tempdir| manifest_path = File::join(tempdir, "temp-migration.manifest.xml") download_manifest(src_s3_conn, bucket, keyprefix+manifest_name, manifest_path, user_pk_path, retry_stuff) ManifestMigrator.new().migrate_manifest(manifest_path, user_pk_path, user_cert_path, user, pass, use_mapping, kernel_id, ramdisk_id, region, true) get_part_filenames(manifest_path, user_cert_path).each do |part| $stdout.puts("Copying '#{part}'...") copy_part(dest_s3_conn, bucket, keyprefix, dest_bucket, dest_keyprefix, part, acl, retry_stuff) end upload_manifest(dest_s3_conn, dest_keyprefix+manifest_name, manifest_path, dest_bucket, tempdir, acl, retry_stuff) end end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() MIGRATE_BUNDLE_MANUAL end def get_name() MIGRATE_BUNDLE_NAME end def main(p) migrate_bundle(p.s3_url, p.bucket, p.keyprefix, p.dest_bucket, p.dest_keyprefix, p.manifest_name, p.user_pk_path, p.user_cert_path, p.user, p.pass, p.location, p.kernel_id, p.ramdisk_id, p.acl, p.retry, p.use_mapping, p.region) $stdout.puts("\nYour new bundle is in S3 at the following location:") $stdout.puts("#{p.dest_bucket}/#{p.dest_keyprefix}#{p.manifest_name}") $stdout.puts("Please register it using your favorite EC2 client.") end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 BundleMigrator.new().run(MigrateBundleParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/uploadbundle.rb0000644000000000000000000002724612050102225021005 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/common/s3support' require 'ec2/amitools/uploadbundleparameters' require 'uri' require 'ec2/amitools/instance-data' require 'ec2/amitools/manifestv20071010' require 'rexml/document' require 'digest/md5' require 'base64' require 'ec2/amitools/tool_base' require 'ec2/amitools/region' #------------------------------------------------------------------------------# UPLOAD_BUNDLE_NAME = 'ec2-upload-bundle' UPLOAD_BUNDLE_MANUAL =<acl, "content-md5"=>md5}) return rescue EC2::Common::HTTP::Error::PathInvalid => e raise FileNotFound(file) rescue => e raise TryFailed.new("Failed to upload \"#{file}\": #{e.message}") end end end #----------------------------------------------------------------------------# def get_md5(file) Base64::encode64(Digest::MD5::digest(File.open(file) { |f| f.read })).strip end #----------------------------------------------------------------------------# # # Availability zone names are generally in the format => ${REGION}${ZONENUMBER}. # Examples being us-east-1b, us-east-1c, etc. # def get_availability_zone() instance_data = EC2::InstanceData.new instance_data.availability_zone end #----------------------------------------------------------------------------# # Return a list of bundle part filename and part number tuples from the manifest. def get_part_info(manifest) parts = manifest.ami_part_info_list.map do |part| [part['filename'], part['index']] end parts.sort end #------------------------------------------------------------------------------# def uri2string(uri) s = "#{uri.scheme}://#{uri.host}:#{uri.port}#{uri.path}" # Remove the trailing '/'. return (s[-1..-1] == "/" ? s[0..-2] : s) end #------------------------------------------------------------------------------# # Get the bucket's location. def get_bucket_location(s3_conn, bucket) begin response = s3_conn.get_bucket_location(bucket) rescue EC2::Common::HTTP::Error::Retrieve => e if e.code == 404 # We have a "Not found" S3 response, which probably means the bucket doesn't exist. return nil end raise e end $stdout.puts "check_bucket_location response: #{response.body}" if @debug and response.text? docroot = REXML::Document.new(response.body).root bucket_location = REXML::XPath.first(docroot, '/LocationConstraint').text bucket_location ||= :unconstrained end #------------------------------------------------------------------------------# # Check if the bucket exists and is in an appropriate location. def check_bucket_location(bucket, bucket_location, location) if bucket_location.nil? # The bucket does not exist. Safe, but we need to create it. return false end if location.nil? # The bucket exists and we don't care where it is. return true end if location != bucket_location # The bucket isn't where we want it. This is a problem. raise BucketLocationError.new(bucket, location, bucket_location) end # The bucket exists and is in the right place. return true end #------------------------------------------------------------------------------# # Create the specified bucket if it does not exist. def create_bucket(s3_conn, bucket, bucket_location, location, retry_create) begin if check_bucket_location(bucket, bucket_location, location) return true end $stdout.puts "Creating bucket..." options = {'Content-Length' => '0'} retry_s3(retry_create) do error = "Could not create or access bucket #{bucket}" begin rsp = s3_conn.create_bucket(bucket, location == :unconstrained ? nil : location, options) rescue EC2::Common::HTTP::Error::Retrieve => e error += ": server response #{e.message} #{e.code}" raise TryFailed.new(e.message) rescue RuntimeError => e error += ": error message #{e.message}" raise e end end end end #------------------------------------------------------------------------------# # If we return true, we have a v2-compliant name. # If we return false, we wish to use a bad name. # Otherwise we quietly wander off to die in peace. def check_bucket_name(bucket) if EC2::Common::S3Support::bucket_name_s3_v2_safe?(bucket) return true end message = "The specified bucket is not S3 v2 safe (see S3 documentation for details):\n#{bucket}" if warn_confirm(message) # Assume the customer knows what he's doing. return false else # We've been asked to stop, so quietly wander off to die in peace. raise EC2StopExecution.new() end end # force v1 S3 addressing when using govcloud endpoint def check_govcloud_override(s3_url) if s3_url =~ /s3-us-gov-west-1/ false else true end end #------------------------------------------------------------------------------# def get_region() zone = get_availability_zone() if zone.nil? return nil end # assume region names do not have a common naming scheme. Therefore we manually go through all known region names AwsRegion.regions.each do |region| match = zone.match(region) if not match.nil? return region end end nil end # This is very much a best effort attempt. If in doubt, we don't warn. def cross_region?(location, bucket_location) # If the bucket exists, its S3 location is canonical. s3_region = bucket_location s3_region ||= location s3_region ||= :unconstrained region = get_region() if region.nil? # If we can't get the region, assume we're fine since there's # nothing more we can do. return false end return s3_region != AwsRegion.get_s3_location(region) end #------------------------------------------------------------------------------# def warn_about_migrating() message = ["You are bundling in one region, but uploading to another. If the kernel", "or ramdisk associated with this AMI are not in the target region, AMI", "registration will fail.", "You can use the ec2-migrate-manifest tool to update your manifest file", "with a kernel and ramdisk that exist in the target region.", ].join("\n") unless warn_confirm(message) raise EC2StopExecution.new() end end #------------------------------------------------------------------------------# def get_s3_conn(s3_url, user, pass, method) EC2::Common::S3Support.new(s3_url, user, pass, method, @debug) end #------------------------------------------------------------------------------# # # Get parameters and display help or manual if necessary. # def upload_bundle(url, bucket, keyprefix, user, pass, location, manifest_file, retry_stuff, part, directory, acl, skipmanifest) begin # Get the S3 URL. s3_uri = URI.parse(url) s3_url = uri2string(s3_uri) v2_bucket = check_bucket_name(bucket) and check_govcloud_override(s3_url) s3_conn = get_s3_conn(s3_url, user, pass, (v2_bucket ? nil : :path)) # Get current location and bucket location. bucket_location = get_bucket_location(s3_conn, bucket) # Load manifest. xml = File.open(manifest_file) { |f| f.read } manifest = ManifestV20071010.new(xml) # If in interactive mode, warn when bundling a kernel into our AMI and we are uploading cross-region if interactive? and manifest.kernel_id and cross_region?(location, bucket_location) warn_about_migrating() end # Create storage bucket if required. create_bucket(s3_conn, bucket, bucket_location, location, retry_stuff) # Upload AMI bundle parts. $stdout.puts "Uploading bundled image parts to the S3 bucket #{bucket} ..." get_part_info(manifest).each do |part_info| if part.nil? or (part_info[1] >= part) path = File.join(directory, part_info[0]) upload(s3_conn, bucket, keyprefix + part_info[0], path, acl, retry_stuff) $stdout.puts "Uploaded #{part_info[0]}" else $stdout.puts "Skipping #{part_info[0]}" end end # Encrypt and upload manifest. unless skipmanifest $stdout.puts "Uploading manifest ..." upload(s3_conn, bucket, keyprefix + File::basename(manifest_file), manifest_file, acl, retry_stuff) $stdout.puts "Uploaded manifest." else $stdout.puts "Skipping manifest." end $stdout.puts 'Bundle upload completed.' rescue EC2::Common::HTTP::Error => e $stderr.puts e.backtrace if @debug raise S3Error.new(e.message) end end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() UPLOAD_BUNDLE_MANUAL end def get_name() UPLOAD_BUNDLE_NAME end def main(p) upload_bundle(p.url, p.bucket, p.keyprefix, p.user, p.pass, p.location, p.manifest, p.retry, p.part, p.directory, p.acl, p.skipmanifest) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 BundleUploader.new().run(UploadBundleParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/manifestv20071010.rb0000644000000000000000000003433712050102225021135 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/crypto' require 'ec2/amitools/format' require 'ec2/amitools/xmlutil' require 'pathname' require 'rexml/document' require 'ec2/amitools/xmlbuilder' # Manifest Version 2007-10-10. # Not backwards compatible class ManifestV20071010 VERSION_STRING = '2007-10-10' VERSION = VERSION_STRING.gsub('-','').to_i # Expose the version def self.version VERSION end # AMI part information container. class PartInformation attr_reader :filename # Part's file basename. attr_reader :digest # Part's digest hex encoded. # Initialize with part's _filename_ and _digest_ as byte string. def initialize( filename, digest ) @filename , @digest = filename, digest end end def initialize( xml = nil ) if xml == nil @doc = REXML::Document.new else # Convert to string if necessary. xml = ( xml.kind_of?( IO ) ? xml.read : xml ) @doc = REXML::Document.new( xml ) end end # for debugging only def doc @doc end def mandatory_argument(arg, args) raise "Missing mandatory argument #{arg} for manifest #{VERSION_STRING}" if !args.key?(arg) args[arg] end def optional_argument(arg, args) args[arg] end IMAGE_TYPE_KERNEL = "kernel" # Initialize the manifest with AMI information. # Return +true+ if the initialization was succesful. # Raise an exception on error. def init(args) name = mandatory_argument(:name, args) user = mandatory_argument(:user, args) # The user's account number. arch = mandatory_argument(:arch, args) # Target architecture for AMI. image_type = mandatory_argument(:image_type, args) # Type of image reserved = mandatory_argument(:reserved, args) # Reserved for future use; pass nil. parts = mandatory_argument(:parts, args) # A list of parts filenames and digest pairs. size = mandatory_argument(:size, args) # The size of the AMI in bytes. bundled_size = mandatory_argument(:bundled_size, args) # The size of the bunled AMI in bytes. user_encrypted_key = mandatory_argument(:user_encrypted_key, args) # Hex encoded. ec2_encrypted_key = mandatory_argument(:ec2_encrypted_key, args) # Hex encoded. cipher_algorithm = mandatory_argument(:cipher_algorithm, args) # The cipher algorithm used to encrypted the AMI. user_encrypted_iv = mandatory_argument(:user_encrypted_iv, args) # Hex encoded. ec2_encrypted_iv = mandatory_argument(:ec2_encrypted_iv, args) # Hex encoded. digest = mandatory_argument(:digest, args) # Hex encoded. digest_algorithm = mandatory_argument(:digest_algorithm, args) # The digest algorithm. privkey_filename = mandatory_argument(:privkey_filename, args) # The user's private key filename. # Optional parameters kernel_id = optional_argument(:kernel_id, args) # Optional default kernel image id ramdisk_id = optional_argument(:ramdisk_id, args) # Optional default ramdisk image id product_codes = optional_argument(:product_codes, args) # Optional array of product codes (strings) ancestor_ami_ids = optional_argument(:ancestor_ami_ids, args) # Optional array of ancestor ami ids (strings) bdm = optional_argument(:block_device_mapping, args) ||{} # Optional hash of block device mappings(strings) bundler_name = optional_argument(:bundler_name, args) bundler_version = optional_argument(:bundler_version, args) bundler_release = optional_argument(:bundler_release, args) # Conditional parameters kernel_name = (image_type == IMAGE_TYPE_KERNEL ? mandatory_argument(:kernel_name, args) : nil) # Name of the kernel in the image # Check reserved parameters are nil raise ArgumentError.new( "reserved parameters not nil" ) unless reserved.nil? # Check non-String parameter types. raise ArgumentError.new( "parts parameter type invalid" ) unless parts.is_a? Array # XML document. @doc = REXML::Document.new @doc << REXML::XMLDecl.new # Yeah... the way we used to do this really sucked - manually building up REXML::Element and inserting them into # parent nodes. So I've reinvented the wheel and done a baby xpath xml builder kinda thing # Makes it much easier (from a code point of view) to build up xml docs. Probably less efficient in machine terms but c'mon # if we cared about that we wouldn't be using ruby. builder = XMLBuilder.new(@doc) # version - indicate the manifest version. builder['/manifest/version'] = VERSION_STRING # bundler information builder['/manifest/bundler/name'] = bundler_name builder['/manifest/bundler/version'] = bundler_version builder['/manifest/bundler/release'] = bundler_release # machine_configuration - the target hardware description of the AMI. builder['/manifest/machine_configuration/architecture'] = arch bdm.keys.sort.each_with_index do |key, index| builder["/manifest/machine_configuration/block_device_mapping/mapping[#{index}]/virtual"] = key builder["/manifest/machine_configuration/block_device_mapping/mapping[#{index}]/device"] = bdm[key] end builder['/manifest/machine_configuration/kernel_id'] = kernel_id builder['/manifest/machine_configuration/ramdisk_id'] = ramdisk_id (product_codes || []).each_with_index do |product_code, index| builder["/manifest/machine_configuration/product_codes/product_code[#{index}]"] = product_code end # image - the image element. builder['manifest/image/name'] = name # user - the user's AWS access key ID. builder['/manifest/image/user'] = user builder['/manifest/image/type'] = image_type # The name of the kernel in the image. Only applicable to kernel images. builder['/manifest/image/kernel_name'] = kernel_name # ancestry - the parent ami ids (ancestor_ami_ids || []).each_with_index do |ancestor_ami_id, index| builder["/manifest/image/ancestry/ancestor_ami_id[#{index}]"] = ancestor_ami_id end # digest - the digest of the AMI. builder['/manifest/image/digest'] = digest builder['/manifest/image/digest/@algorithm'] = digest_algorithm # size - the size of the uncompressed AMI. builder['/manifest/image/size'] = size.to_s # bundled size - the size of the bundled AMI. builder['/manifest/image/bundled_size'] = bundled_size.to_s # ec2 encrypted key element. builder['/manifest/image/ec2_encrypted_key'] = ec2_encrypted_key builder['/manifest/image/ec2_encrypted_key/@algorithm'] = cipher_algorithm # user encrypted key element. builder['/manifest/image/user_encrypted_key'] = user_encrypted_key builder['/manifest/image/user_encrypted_key/@algorithm'] = cipher_algorithm # ec2 encrypted iv element. builder['/manifest/image/ec2_encrypted_iv'] = ec2_encrypted_iv # user encrypted iv element. builder['/manifest/image/user_encrypted_iv'] = user_encrypted_iv # parts - list of the image parts. builder['/manifest/image/parts/@count'] = parts.size parts.each_with_index do |part, index| # Add image part element for each image part. builder["/manifest/image/parts/part[#{index}]/@index"] = index builder["/manifest/image/parts/part[#{index}]/filename"] = part[0] builder["/manifest/image/parts/part[#{index}]/digest"] = Format::bin2hex(part[1]) builder["/manifest/image/parts/part[#{index}]/digest/@algorithm"] = digest_algorithm builder["/manifest/image/parts/part[#{index}]/@index"] = index end # Sign the manifest. sign(privkey_filename) return true end def self::version20071010?(xml) doc = REXML::Document.new(xml) version = REXML::XPath.first(doc.root, 'version') return (version and version.text and version.text == VERSION_STRING) end # Get the kernel_name def kernel_name() return get_element_text_or_nil('image/kernel_name') end # Get the default kernel_id def kernel_id() return get_element_text_or_nil('machine_configuration/kernel_id') end # Get the default ramdisk id def ramdisk_id() return get_element_text_or_nil('machine_configuration/ramdisk_id') end # Get the default product codes def product_codes() product_codes = [] REXML::XPath.each(@doc, '/manifest/machine_configuration/product_codes/product_code') do |product_code| product_codes << product_code.text end product_codes end # Get the manifest's ancestry def ancestor_ami_ids() ancestor_ami_ids = [] REXML::XPath.each(@doc, '/manifest/image/ancestry/ancestor_ami_id') do |node| ancestor_ami_ids << node.text unless (node.text.nil? or node.text.empty?) end ancestor_ami_ids end # Return the AMI's digest hex encoded. def digest() return get_element_text( 'image/digest' ) end def name() return get_element_text( 'image/name' ) end # The ec2 encrypted key hex encoded. def ec2_encrypted_key() return get_element_text('image/ec2_encrypted_key' ) end # The user encrypted key hex encoded. def user_encrypted_key() return get_element_text( 'image/user_encrypted_key' ) end # The ec2 encrypted initialization vector hex encoded. def ec2_encrypted_iv() return get_element_text( 'image/ec2_encrypted_iv' ) end # The user encrypted initialization vector hex encoded. def user_encrypted_iv() return get_element_text( 'image/user_encrypted_iv' ) end # Get digest algorithm used. def digest_algorithm() return REXML::XPath.first(@doc.root, 'image/digest/@algorithm').to_s end # Get cipher algorithm used. def cipher_algorithm() return REXML::XPath.first(@doc.root, 'image/ec2_encrypted_key/@algorithm').to_s end # Retrieve a list of AMI bundle parts info. Each element is a hash # with the following elements: # * 'digest' # * 'filename' # * 'index' def ami_part_info_list parts = Array.new REXML::XPath.each( @doc.root,'image/parts/part' ) do |part| index = part.attribute( 'index' ).to_s.to_i filename = REXML::XPath.first( part, 'filename' ).text digest = REXML::XPath.first( part, 'digest' ).text parts << { 'digest'=>digest, 'filename'=>filename, 'index'=>index } end return parts end # A list of PartInformation instances representing the AMI parts. def parts() parts = [] REXML::XPath.each( @doc.root,'image/parts/part' ) do |part| index = part.attribute( 'index' ).to_s.to_i filename = REXML::XPath.first( part, 'filename' ).text digest = Format::hex2bin( REXML::XPath.first( part, 'digest' ).text ) parts[index] = PartInformation.new( filename, digest ) end return parts end # Get the block device mapping as a map. def block_device_mapping() bdm = {} REXML::XPath.each(@doc.root,'machine_configuration/block_device_mapping/mapping/') do |mapping| virtual = REXML::XPath.first(mapping, 'virtual').text device = REXML::XPath.first(mapping, 'device').text bdm[virtual] = device end bdm end # Return the size of the AMI. def size() return get_element_text( 'image/size' ).to_i() end # Return the (optional) architecture of the AMI. def arch() return get_element_text_or_nil('machine_configuration/architecture') end # Return the bundled size of the AMI. def bundled_size() return get_element_text( 'image/bundled_size' ).to_i end # Return the bundler name. def bundler_name() return get_element_text('bundler/name') end # Return the bundler version. def bundler_version() return get_element_text('bundler/version') end # Return the bundler release. def bundler_release() return get_element_text('bundler/release') end # Sign the manifest. If it is already signed, the signature and certificate # will be replaced def sign( privkey_filename ) unless privkey_filename.kind_of? String and File::exist?( privkey_filename ) raise ArgumentError.new( "privkey_filename parameter invalid" ) end # Get the XML for and elements and sign them. machine_configuration_xml = XMLUtil.get_xml( @doc.to_s, 'machine_configuration' ) || "" image_xml = XMLUtil.get_xml( @doc.to_s, 'image' ) sig = Crypto::sign( machine_configuration_xml + image_xml, privkey_filename ) # Create the signature and certificate elements. XMLBuilder.new(@doc)['/manifest/signature'] = Format::bin2hex(sig) end # Return the signature def signature get_element_text('signature') end # Verify the signature def authenticate(cert) machine_configuration_xml = XMLUtil.get_xml( @doc.to_s, 'machine_configuration' ) || "" image_xml = XMLUtil.get_xml( @doc.to_s, 'image' ) pubkey = Crypto::cert2pubkey(cert) Crypto::authenticate(machine_configuration_xml + image_xml, Format::hex2bin(signature), pubkey) end # Return the manifest as an XML string. def to_s() return @doc.to_s end def user() return get_element_text('image/user') end def image_type() return get_element_text('image/type') end def version() return get_element_text('version').gsub('-','').to_i end private def get_element_text_or_nil(xpath) element = REXML::XPath.first(@doc.root, xpath) return element.text if element return nil end def get_element_text(xpath) element = REXML::XPath.first(@doc.root, xpath) unless element raise "invalid AMI manifest, #{xpath} element not present" end unless element.text raise "invalid AMI manifest, #{xpath} element empty" end return element.text end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/unbundle.rb0000644000000000000000000000701612050102225020134 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/format' require 'ec2/amitools/manifestv3' require 'ec2/amitools/unbundleparameters' require 'ec2/platform/current' require 'ec2/amitools/tool_base' UNBUNDLE_NAME = 'ec2-unbundle' UNBUNDLE_MANUAL =< e $stderr.puts e.message end # Verify digest. unless manifest.digest == digest raise "invalid digest, expected #{manifest.digest} received #{digest}" end puts "Unbundle complete." return 0 ensure File::delete( digest_pipe ) if (digest_pipe && File::exist?( digest_pipe )) end end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() UNBUNDLE_MANUAL end def get_name() UNBUNDLE_NAME end def main(p) unbundle(p) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 Unbundler.new().run(UnbundleParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/minimalec2.rb0000644000000000000000000000770512050102225020345 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'base64' require 'cgi' require 'openssl' require 'digest/sha1' require 'net/https' require 'rexml/document' require 'time' require 'ec2/amitools/version' module EC2 class EC2Client attr_accessor :verbose attr_accessor :aws_access_key_id attr_accessor :aws_secret_access_key attr_accessor :http def parse_url(url) bits = url.split(":") secure = {"https"=>true, "http"=>false}[bits[0]] port = secure ? 443 : 80 port = Integer(bits[2]) if bits.size > 2 server = bits[1][2..-1] [server, port, secure] end def initialize(akid, secretkey, url) @aws_access_key_id = akid @aws_secret_access_key = secretkey server, port, is_secure = parse_url(url) @http = Net::HTTP.new(server, port) @http.use_ssl = is_secure @verbose = false end def pathlist(key, arr) params = {} arr.each_with_index do |value, i| params["#{key}.#{i+1}"] = value end params end def describe_regions(regionNames=[]) params = pathlist("regionName", regionNames) make_request("DescribeRegions", params) end def describe_images(imageIds=[], kwargs={}) params = pathlist("ImageId", imageIds) params.merge!(pathlist("Owner", kwargs[:owners])) if kwargs[:owners] params.merge!(pathlist("ExecutableBy", kwargs[:executableBy])) if kwargs[:executableBy] make_request("DescribeImages", params) end def make_request(action, params, data='') resp = nil @http.start do params.merge!({ "Action"=>action, "SignatureVersion"=>"1", "AWSAccessKeyId"=>@aws_access_key_id, "Version"=> "2008-12-01", "Timestamp"=>Time.now.getutc.iso8601, }) p params if @verbose canonical_string = params.sort_by { |param| param[0].downcase }.map { |param| param.join }.join puts canonical_string if @verbose sig = encode(@aws_secret_access_key, canonical_string) path = "?" + params.sort.collect do |param| CGI::escape(param[0]) + "=" + CGI::escape(param[1]) end.join("&") + "&Signature=" + sig puts path if @verbose req = Net::HTTP::Get.new("/#{path}") # ruby will automatically add a random content-type on some verbs, so # here we add a dummy one to 'supress' it. change this logic if having # an empty content-type header becomes semantically meaningful for any # other verb. req['Content-Type'] ||= '' req['User-Agent'] = 'ec2-migrate-manifest #{PKG_VERSION}-#{PKG_RELEASE}' data = nil unless req.request_body_permitted? resp = @http.request(req, data) end REXML::Document.new(resp.body) end # Encodes the given string with the aws_secret_access_key, by taking the # hmac-sha1 sum, and then base64 encoding it. Optionally, it will also # url encode the result of that to protect the string if it's going to # be used as a query string parameter. def encode(aws_secret_access_key, str, urlencode=true) digest = OpenSSL::Digest::Digest.new('sha1') b64_hmac = Base64.encode64(OpenSSL::HMAC.digest(digest, aws_secret_access_key, str)).strip if urlencode return CGI::escape(b64_hmac) else return b64_hmac end end end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/deletebundle.rb0000644000000000000000000001525112050102225020754 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/crypto' require 'ec2/amitools/exception' require 'ec2/amitools/deletebundleparameters' require 'net/https' require 'rexml/document' require 'tempfile' require 'uri' require 'ec2/common/s3support' require 'ec2/amitools/tool_base' DELETE_BUNDLE_NAME = 'ec2-delete-bundle' #------------------------------------------------------------------------------# DELETE_BUNDLE_MANUAL=< e raise TryFailed.new("Failed to delete \"#{key}\": #{e.message}") end end end #----------------------------------------------------------------------------# # Return a list of bundle part filenames from the manifest. def get_part_filenames(manifest) parts = [] manifest_doc = REXML::Document.new(manifest).root REXML::XPath.each(manifest_doc, 'image/parts/part/filename/text()') do |part| parts << part.to_s end return parts end #------------------------------------------------------------------------------# def uri2string(uri) s = "#{uri.scheme}://#{uri.host}:#{uri.port}#{uri.path}" # Remove the trailing '/'. return (s[-1..-1] == "/" ? s[0..-2] : s) end #------------------------------------------------------------------------------# def get_file_list_from_s3(bucket, keyprefix, prefix) s3prefix = keyprefix+prefix files_to_delete = [] response = @s3_conn.list_bucket(bucket, s3prefix) unless response.success? raise "unable to list contents of bucket #{bucket}: HTTP #{response.code} response: #{response.body}" end REXML::XPath.each(REXML::Document.new(response.body), "//Key/text()") do |entry| entry = entry.to_s if entry[0,s3prefix.length] == s3prefix test_str = entry[(s3prefix.length)..-1] if (test_str =~ /^\.part\.[0-9]+$/ or test_str =~ /^\.manifest(\.xml)?$/) files_to_delete << entry[(keyprefix.length)..-1] end end end files_to_delete end #------------------------------------------------------------------------------# def make_s3_connection(s3_url, user, pass, method) EC2::Common::S3Support.new(s3_url, user, pass, method, @debug) end #------------------------------------------------------------------------------# def delete_bundle(url, bucket, keyprefix, user, pass, manifest, prefix, yes, clear, retry_stuff) begin # Get the S3 URL. s3_uri = URI.parse(url) s3_url = uri2string(s3_uri) retry_delete = retry_stuff v2_bucket = EC2::Common::S3Support::bucket_name_s3_v2_safe?(bucket) @s3_conn = make_s3_connection(s3_url, user, pass, (v2_bucket ? nil : :path)) files_to_delete = [] if manifest # Get list of files to delete from the AMI manifest. xml = String.new manifest_path = manifest File.open(manifest_path) { |f| xml << f.read } files_to_delete << File::basename(manifest) get_part_filenames( xml ).each do |part_info| files_to_delete << part_info end else files_to_delete = get_file_list_from_s3(bucket, keyprefix, prefix) end if files_to_delete.empty? $stdout.puts "No files to delete." else $stdout.puts "Deleting files:" files_to_delete.each { |file| $stdout.puts(" - #{file}") } continue = yes unless continue begin $stdout.print "Continue [y/N]: " $stdout.flush Timeout::timeout(PROMPT_TIMEOUT) do continue = gets.strip =~ /^y/i end rescue Timeout::Error $stdout.puts "\nNo response given, skipping the files." continue = false end end if continue files_to_delete.each do |file| delete(bucket, keyprefix+file, retry_delete) $stdout.puts "Deleted #{file}" end end end if clear $stdout.puts "Attempting to delete bucket #{bucket}..." @s3_conn.delete(bucket) end rescue EC2::Common::HTTP::Error => e $stderr.puts e.backtrace if @debug raise S3Error.new(e.message) end $stdout.puts "#{DELETE_BUNDLE_NAME} complete." end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() DELETE_BUNDLE_MANUAL end def get_name() DELETE_BUNDLE_NAME end def main(p) delete_bundle(p.url, p.bucket, p.keyprefix, p.user, p.pass, p.manifest, p.prefix, p.yes, p.clear, p.retry) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 BundleDeleter.new().run(DeleteBundleParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/tool_base.rb0000644000000000000000000001363312050102225020271 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. module AMIToolExceptions # All fatal errors should inherit from this. class EC2FatalError < RuntimeError attr_accessor :code def initialize(code, msg) super(msg) @code = code end end class FileNotFound < EC2FatalError def initialize(path) super(2, "File not found: #{path}") end end class S3Error < EC2FatalError def initialize(msg) super(3, "Error talking to S3: #{msg}") end end class PromptTimeout < EC2FatalError def initialize(msg=nil) message = "Timed out waiting for user input" message += ": #{msg}" unless msg.nil? super(5, message) end end # This is more for flow control than anything else. # Raising it should terminate execution, but not print an error. class EC2StopExecution < RuntimeError attr_accessor :code def initialize(code=0) super() @code = code end end class TryFailed < RuntimeError end end class AMITool include AMIToolExceptions PROMPT_TIMEOUT = 30 MAX_TRIES = 5 BACKOFF_PERIOD = 5 #------------------------------------------------------------------------------# # Methods to override in subclasses #------------------------------------------------------------------------------# def get_manual() # We have to get the manual text into here. raise "NotImplemented: get_manual()" end def get_name() # We have to get the tool name into here. raise "NotImplemented: get_name()" end def main(params) # Main entry point. raise "NotImplemented: main()" end #------------------------------------------------------------------------------# # Utility methods #------------------------------------------------------------------------------# # Display a message (without appending a newline) and ask for a response. # Returns user response or nil if interactivity is not desired. # Raises exception on timeout. def interactive_prompt(message, name=nil) return nil unless interactive? begin $stdout.print(message) $stdout.flush Timeout::timeout(PROMPT_TIMEOUT) do return gets end rescue Timeout::Error raise PromptTimeout.new(name) end end #------------------------------------------------------------------------------# # Display a message on stderr. # If interactive, asks for confirmation (yes/no). # Returns true if in batch mode or user agrees, false if user disagrees. # Raises exception on timeout. def warn_confirm(message) $stderr.puts(message) $stderr.flush return true unless interactive? response = interactive_prompt("Are you sure you want to continue? [y/N]") if response =~ /^[Yy]/ return true end return false end #----------------------------------------------------------------------------# def retry_s3(retrying=true) tries = 0 while true tries += 1 begin result = yield return result rescue TryFailed => e $stderr.puts e.message if retrying and tries < MAX_TRIES $stdout.puts "Retrying in #{BACKOFF_PERIOD}s ..." else raise EC2FatalError.new(3, e.message) end end end end #------------------------------------------------------------------------------# # Standard behaviour #------------------------------------------------------------------------------# def handle_early_exit_parameters(params) if params.version puts get_name() + " " + params.version_copyright_string() return end if params.show_help puts params.help return end if params.manual puts get_manual() return end end #------------------------------------------------------------------------------# def interactive? @interactive end #------------------------------------------------------------------------------# def get_parameters(params_class) # Parse the parameters and die on errors. # Assume that if we're parsing parameters, it's safe to exit. begin params = params_class.new(ARGV) rescue StandardError => e $stderr.puts e.message $stderr.puts "Try '#{get_name} --help'" exit 1 end # Deal with help, verion, etc. if params.early_exit? handle_early_exit_parameters(params) exit 0 end # Some general flags that we want to set @debug = params.debug @interactive = params.interactive? # Finally, return the leftovers. params end #------------------------------------------------------------------------------# def run(params_class) # We want to be able to reuse bits without having to parse # parameters, so run() is not called from the constructor. begin params = get_parameters(params_class) main(params) rescue AMIToolExceptions::EC2StopExecution => e # We've been asked to stop. exit e.code rescue AMIToolExceptions::PromptTimeout => e $stderr.puts e.message exit e.code rescue AMIToolExceptions::EC2FatalError => e $stderr.puts "ERROR: #{e.message}" puts e.backtrace if @debug exit e.code rescue Interrupt => e $stderr.puts "\n#{get_name} interrupted." puts e.backtrace if @debug exit 255 rescue => e $stderr.puts "ERROR: #{e.message}" puts e.inspect if @debug puts e.backtrace if @debug exit 254 end end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/parameter_exceptions.rb0000644000000000000000000000211412050102225022533 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. module ParameterExceptions class Error < RuntimeError end class MissingMandatory < Error def initialize(name) super("missing mandatory parameter: #{name}") end end class InvalidCombination < Error def initialize(name1, name2) super("#{name1} and #{name2} may not both be provided") end end class InvalidValue < Error def initialize(name, value, msg=nil) message = "#{name} has invalid value '#{value.to_s}'" message += ": #{msg}" unless msg.nil? super(message) end end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundleparameters.rb0000644000000000000000000001070212050102225021651 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/parameters_base' require 'timeout' require 'ec2/platform/current' require 'ec2/amitools/syschecks' # The Bundle command line parameters. class BundleParameters < ParametersBase include EC2::Platform::Current::Constants SUPPORTED_ARCHITECTURES = ['i386', 'x86_64'] USER_DESCRIPTION = "The user's EC2 user ID (Note: AWS account number, NOT Access Key ID)." HELP_DESCRIPTION = "Display this help message and exit." MANUAL_DESCRIPTION = "Display the user manual and exit." DESTINATION_DESCRIPTION = "The directory to create the bundle in. Defaults to '#{Bundling::DESTINATION}'." DEBUG_DESCRIPTION = "Display debug messages." EC2_CERT_PATH_DESCRIPTION = ['The path to the EC2 X509 public key certificate bundled into the AMI.', "Defaults to '#{Bundling::EC2_X509_CERT}'."] ARCHITECTURE_DESCRIPTION = "Specify target architecture. One of #{SUPPORTED_ARCHITECTURES.inspect}" BATCH_DESCRIPTION = "Run in batch mode. No interactive prompts." PRODUCT_CODES_DESCRIPTION = ['Default product codes attached to the image at registration time.', 'Comma separated list of product codes.'] SIZE_CHECKS_DESCRIPTION = 'If set, disables size checks on bundled artifacts.' VERSION_DESCRIPTION = "Display the version and copyright notice and then exit." attr_accessor :user_pk_path, :user_cert_path, :user, :destination, :ec2_cert_path, :debug, :show_help, :manual, :arch, :batch_mode, :size_checks, :product_codes PROMPT_TIMEOUT = 30 #----------------------------------------------------------------------------# def mandatory_params() on('-c', '--cert PATH', String, USER_CERT_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--cert') @user_cert_path = path end on('-k', '--privatekey PATH', String, USER_PK_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--privatekey') @user_pk_path = path end on('-u', '--user USER', String, USER_ACCOUNT_DESCRIPTION) do |user| # Remove hyphens from the Account ID as presented in AWS portal. @user = user.gsub("-", "") # Validate the account ID looks correct (users often provide us with their akid or secret key) unless (@user =~ /\d{12}/) raise InvalidValue.new('--user', @user, "the user ID should consist of 12 digits (optionally hyphenated); this should not be your Access Key ID") end end end #----------------------------------------------------------------------------# def optional_params() on('-d', '--destination PATH', String, DESTINATION_DESCRIPTION) do |path| assert_directory_exists(path, '--destination') @destination = path end on('--ec2cert PATH', String, *BundleParameters::EC2_CERT_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--ec2cert') @ec2_cert_path = path end on('-r', '--arch ARCHITECTURE', String, ARCHITECTURE_DESCRIPTION) do |arch| @arch = arch end on('--productcodes PRODUCT_CODES', String, *PRODUCT_CODES_DESCRIPTION) do |pc| @product_codes = pc end on('--no-size-checks', SIZE_CHECKS_DESCRIPTION ) do |o| @size_checks = o end end #----------------------------------------------------------------------------# def validate_params() raise MissingMandatory.new('--cert') unless @user_cert_path raise MissingMandatory.new('--privatekey') unless @user_pk_path raise MissingMandatory.new('--user') unless @user end #----------------------------------------------------------------------------# def set_defaults() @destination ||= Bundling::DESTINATION @ec2_cert_path ||= Bundling::EC2_X509_CERT @exclude ||= [] @size_checks = true end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundlevol.rb0000644000000000000000000001626312050102225020316 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/bundle' require 'ec2/amitools/bundlevolparameters' require 'ec2/platform/current' require 'ec2/amitools/syschecks' require 'ec2/amitools/bundle_base' BUNDLE_VOL_NAME = 'ec2-bundle-vol' BUNDLE_VOL_MANUAL=< p.kernel_id, :ramdisk_id => p.ramdisk_id, :product_codes => p.product_codes, :ancestor_ami_ids => p.ancestor_ami_ids, :block_device_mapping => p.block_device_mapping } Bundle.bundle_image(image_file, p.user, p.arch, Bundle::ImageType::VOLUME, p.destination, p.user_pk_path, p.user_cert_path, p.ec2_cert_path, nil, # prefix optional_args, @debug, p.inherit) $stdout.puts("#{BUNDLE_VOL_NAME} complete.") end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() BUNDLE_VOL_MANUAL end def get_name() BUNDLE_VOL_NAME end def main(p) bundle_vol(p) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 VolBundler.new().run(BundleVolParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/deletebundleparameters.rb0000644000000000000000000000461212050102225023037 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/s3toolparameters' #------------------------------------------------------------------------------# class DeleteBundleParameters < S3ToolParameters MANIFEST_DESCRIPTION = "The path to the unencrypted manifest file." PREFIX_DESCRIPTION = "The bundled AMI part filename prefix." RETRY_DESCRIPTION = "Automatically retry failed deletes. Use with caution." YES_DESCRIPTION = "Automatically answer 'y' without asking." CLEAR_DESCRIPTION = "Delete the bucket if empty. Not done by default." attr_accessor :manifest, :prefix, :retry, :yes, :clear #----------------------------------------------------------------------------# def mandatory_params() super() end #----------------------------------------------------------------------------# def optional_params() super() on('-m', '--manifest PATH', String, MANIFEST_DESCRIPTION) do |manifest| assert_file_exists(manifest, '--manifest') @manifest = manifest end on('-p', '--prefix PREFIX', String, PREFIX_DESCRIPTION) do |prefix| assert_good_key(prefix, '--prefix') @prefix = prefix end on('--clear', CLEAR_DESCRIPTION) do @clear = true end on('--retry', RETRY_DESCRIPTION) do @retry = true end on('-y', '--yes', YES_DESCRIPTION) do @yes = true end end #----------------------------------------------------------------------------# def validate_params() super() raise MissingMandatory.new('--manifest or --prefix') unless @manifest or @prefix raise InvalidCombination.new('--prefix', '--manifest') if (@prefix and @manifest) end #----------------------------------------------------------------------------# def set_defaults() super() @clear ||= false end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/downloadbundle.rb0000644000000000000000000001267412050102225021327 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/crypto' require 'ec2/common/s3support' require 'ec2/amitools/downloadbundleparameters' require 'ec2/amitools/exception' require 'ec2/amitools/manifestv20071010' require 'getoptlong' require 'net/http' require 'rexml/document' require 'ec2/amitools/tool_base' # Download AMI downloads the specified AMI from S3. #------------------------------------------------------------------------------# DOWNLOAD_BUNDLE_NAME = 'ec2-download-bundle' DOWNLOAD_BUNDLE_MANUAL =< e raise TryFailed.new("Failed to download \"#{file}\": #{e.message}") end end end #----------------------------------------------------------------------------# def get_part_filenames(manifest_xml) manifest = ManifestV20071010.new(manifest_xml) manifest.parts.collect { |part| part.filename }.sort end #----------------------------------------------------------------------------# def uri2string( uri ) s = "#{uri.scheme}://#{uri.host}:#{uri.port}#{uri.path}" # Remove the trailing '/'. return ( s[-1..-1] == "/" ? s[0..-2] : s ) end #----------------------------------------------------------------------------# def make_s3_connection(s3_url, user, pass, bucket) s3_uri = URI.parse(s3_url) s3_url = uri2string(s3_uri) v2_bucket = EC2::Common::S3Support::bucket_name_s3_v2_safe?(bucket) EC2::Common::S3Support.new(s3_url, user, pass, (v2_bucket ? nil : :path), @debug) end #----------------------------------------------------------------------------# # Main method. def download_bundle(url, user, pass, bucket, keyprefix, directory, manifest, privatekey, retry_stuff) begin s3_conn = make_s3_connection(url, user, pass, bucket) # Download and decrypt manifest. manifest_path = File.join(directory, manifest) manifest_xml = download_manifest(s3_conn, bucket, keyprefix+manifest, manifest_path, privatekey, retry_stuff) # Download AMI parts. get_part_filenames(manifest_xml).each do |filename| download_part(s3_conn, bucket, keyprefix+filename, File::join(directory, filename), retry_stuff) $stdout.puts "Downloaded #{filename} from #{bucket}" end rescue EC2::Common::HTTP::Error => e $stderr.puts e.backtrace if @debug raise S3Error.new(e.message) end end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() DOWNLOAD_BUNDLE_MANUAL end def get_name() DOWNLOAD_BUNDLE_NAME end def main(p) download_bundle(p.url, p.user, p.pass, p.bucket, p.keyprefix, p.directory, p.manifest, p.privatekey, p.retry) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 BundleDownloader.new().run(DownloadBundleParameters) end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/showversion.rb0000644000000000000000000000116512050102225020705 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/version' puts EC2Version::version_copyright_string() ec2-ami-tools-1.4.0.9/lib/ec2/amitools/defaults.rb0000644000000000000000000000123412050102225020123 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. EC2_HOME = ENV['EC2_AMITOOL_HOME'] || ENV['EC2_HOME'] EC2_X509_CERT = '#{EC2_HOME}/etc/ec2/amitools/cert-ec2.pem' ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundlemachineparameters.rb0000644000000000000000000000617612050102225023210 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/bundleparameters' # The Bundle command line parameters. class BundleMachineParameters < BundleParameters KERNEL_DESCRIPTION = "Id of the default kernel to launch the AMI with." RAMDISK_DESCRIPTION = "Id of the default ramdisk to launch the AMI with." ANCESTOR_AMI_IDS_DESCRIPTION = "Lineage of this image. Comma separated list of AMI ids." BDM_DESCRIPTION = ['Default block-device-mapping scheme to launch the AMI with. This scheme', 'defines how block devices may be exposed to an EC2 instance of this AMI', 'if the instance-type of the instance is entitled to the specified device.', 'The scheme is a comma-separated list of key=value pairs, where each key', 'is a "virtual-name" and each value, the corresponding native device name', 'desired. Possible virtual-names are:', ' - "ami": denotes the root file system device, as seen by the instance.', ' - "root": denotes the root file system device, as seen by the kernel.', ' - "swap": denotes the swap device, if present.', ' - "ephemeralN": denotes Nth ephemeral store; N is a non-negative integer.', 'Note that the contents of the AMI form the root file system. Samples of', 'block-device-mappings are:', ' - "ami=sda1,root=/dev/sda1,ephemeral0=sda2,swap=sda3"', ' - "ami=0,root=/dev/dsk/c0d0s0,ephemeral0=1"' ] attr_accessor :kernel_id, :ramdisk_id, :ancestor_ami_ids, :block_device_mapping def optional_params() super() on( '--kernel ID', KERNEL_DESCRIPTION ) do |id| @kernel_id = id end on( '--ramdisk ID', RAMDISK_DESCRIPTION ) do |id| @ramdisk_id = id end on( '-B', '--block-device-mapping MAPS', String, *BDM_DESCRIPTION ) do |bdm| @block_device_mapping ||= {} raise InvalidValue.new('--block-device-mapping', bdm) if bdm.to_s.empty? bdm.split(',').each do |mapping| raise InvalidValue.new('--block-device-mapping', bdm) unless mapping =~ /^\s*(\S)+\s*=\s*(\S)+\s*$/ virtual, device = mapping.split(/=/) @block_device_mapping[virtual.strip] = device.strip end end end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/version.rb0000644000000000000000000000270712050102225020007 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/manifestv20071010' module EC2Version MANIFEST_CLASS = ManifestV20071010 MANIFEST_VERSION = MANIFEST_CLASS.version PKG_NAME = 'ec2-ami-tools' PKG_VERSION = '1.4' PKG_RELEASE = '0.9' COPYRIGHT_NOTICE = < parts.size.to_s}) index=0 parts.each do |part| # Add image part element for each image part. part_element = REXML::Element.new('part') part_element.add_attribute('index', index.to_s) filename = REXML::Element.new('filename') filename.add_text(part[0]) part_element.add_element(filename) digest = REXML::Element.new('digest') digest.add_attribute('algorithm', digest_algorithm) digest.add_text(Format::bin2hex(part[1])) part_element.add_element(digest) parts_element.add_element(part_element) index+=1 end image.add_element(parts_element) # Sign the manifest. sign(privkey_filename) return true end def self::version20070829?(xml) doc = REXML::Document.new(xml) version = REXML::XPath.first(doc.root, 'version') return true if (version and version.text and version.text == VERSION_STRING) return (version and version.text and version.text == VERSION.to_s) end # Return the AMI's digest hex encoded. def digest() return get_element_text('image/digest') end def name() return get_element_text('image/name') end # The ec2 encrypted key hex encoded. def ec2_encrypted_key() return get_element_text('image/ec2_encrypted_key') end # The user encrypted key hex encoded. def user_encrypted_key() return get_element_text('image/user_encrypted_key') end # The ec2 encrypted initialization vector hex encoded. def ec2_encrypted_iv() return get_element_text('image/ec2_encrypted_iv') end # The user encrypted initialization vector hex encoded. def user_encrypted_iv() return get_element_text('image/user_encrypted_iv') end # Get digest algorithm used. def digest_algorithm() return REXML::XPath.first(@doc.root, 'image/digest/@algorithm').to_s end # Get cipher algorithm used. def cipher_algorithm() return REXML::XPath.first(@doc.root, 'image/ec2_encrypted_key/@algorithm').to_s end # Retrieve a list of AMI bundle parts info. Each element is a hash # with the following elements: # * 'digest' # * 'filename' # * 'index' def ami_part_info_list parts = Array.new REXML::XPath.each(@doc.root,'image/parts/part') do |part| index = part.attribute('index').to_s.to_i filename = REXML::XPath.first(part, 'filename').text digest = REXML::XPath.first(part, 'digest').text parts << { 'digest'=>digest, 'filename'=>filename, 'index'=>index } end return parts end # A list of PartInformation instances representing the AMI parts. def parts() parts = [] REXML::XPath.each(@doc.root,'image/parts/part') do |part| index = part.attribute('index').to_s.to_i filename = REXML::XPath.first(part, 'filename').text digest = Format::hex2bin(REXML::XPath.first(part, 'digest').text) parts[index] = PartInformation.new(filename, digest) end return parts end # Return the size of the AMI. def size() return get_element_text('image/size').to_i() end # Return the (optional) architecture of the AMI. def arch() return get_element_text('machine_configuration/architecture') end # Return the bundled size of the AMI. def bundled_size() return get_element_text('image/bundled_size').to_i end # Return the bundler name. def bundler_name() return get_element_text('bundler/name') end # Return the bundler version. def bundler_version() return get_element_text('bundler/version') end # Return the bundler release. def bundler_release() return get_element_text('bundler/release') end # Sign the manifest. If it is already signed, the signature and certificate # will be replaced def sign(privkey_filename) unless privkey_filename.kind_of? String and File::exist?(privkey_filename) raise ArgumentError.new("privkey_filename parameter invalid") end # Get the XML for and elements and sign them. machine_configuration_xml = XMLUtil.get_xml(@doc.to_s, 'machine_configuration') || "" image_xml = XMLUtil.get_xml(@doc.to_s, 'image') sig = Crypto::sign(machine_configuration_xml + image_xml, privkey_filename) # Create the signature and certificate elements. signature = REXML::Element.new('signature') signature.add_text(Format::bin2hex(sig)) @doc.root.delete_element('signature') @doc.root.add_element(signature) end # Return the signature def signature get_element_text('signature') end # Verify the signature def authenticate(cert) machine_configuration_xml = XMLUtil.get_xml(@doc.to_s, 'machine_configuration') || "" image_xml = XMLUtil.get_xml(@doc.to_s, 'image') pubkey = Crypto::cert2pubkey(cert) Crypto::authenticate(machine_configuration_xml + image_xml, Format::hex2bin(signature), pubkey) end # Return the manifest as an XML string. def to_s() return @doc.to_s end def user() return get_element_text('image/user') end def version() return get_element_text('version').to_i end private def get_element_text(xpath) element = REXML::XPath.first(@doc.root, xpath) unless element raise "invalid AMI manifest, #{xpath} element not present" end unless element.text raise "invalid AMI manifest, #{xpath} element empty" end return element.text end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/parameters_base.rb0000644000000000000000000001104312050102225021450 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'optparse' require 'ec2/amitools/parameter_exceptions' require 'ec2/amitools/version' require 'ec2/common/s3support' class ParametersBase < OptionParser include ParameterExceptions # Descriptions for common parameters: USER_CERT_PATH_DESCRIPTION = "The path to the user's PEM encoded RSA public key certificate file." USER_PK_PATH_DESCRIPTION = "The path to the user's PEM encoded RSA private key file." USER_DESCRIPTION = "The user's AWS access key ID." PASS_DESCRIPTION = "The user's AWS secret access key." USER_ACCOUNT_DESCRIPTION = "The user's EC2 user ID (Note: AWS account number, NOT Access Key ID)." HELP_DESCRIPTION = "Display this help message and exit." MANUAL_DESCRIPTION = "Display the user manual and exit." DEBUG_DESCRIPTION = "Display debug messages." VERSION_DESCRIPTION = "Display the version and copyright notice and then exit." BATCH_DESCRIPTION = "Run in batch mode. No interactive prompts." attr_accessor(:show_help, :manual, :version, :batch_mode, :debug) #------------------------------------------------------------------------------# # Methods to override in subclasses #------------------------------------------------------------------------------# def mandatory_params() # Override this for mandatory parameters end def optional_params() # Override this for optional parameters end def validate_params() # Override this for parameter validation end def set_defaults() # Override this for parameter validation end #------------------------------------------------------------------------------# # Useful utility methods #------------------------------------------------------------------------------# def early_exit?() @show_help or @manual or @version end def interactive?() not (early_exit? or @batch_mode) end def version_copyright_string() EC2Version::version_copyright_string() end #------------------------------------------------------------------------------# # Validation utility methods #------------------------------------------------------------------------------# def assert_exists(path, param) unless File::exist?(path) raise InvalidValue.new(param, path, "File or directory does not exist.") end end def assert_file_exists(path, param) unless (File::exist?(path) and File::file?(path)) raise InvalidValue.new(param, path, "File does not exist or is not a file.") end end def assert_directory_exists(path, param) unless (File::exist?(path) and File::directory?(path)) raise InvalidValue.new(param, path, "Directory does not exist or is not a directory.") end end def assert_option_in(option, choices, param) unless choices.include?(option) raise InvalidValue.new(param, option) end end def assert_good_key(key, param) if key.include?("/") raise InvalidValue.new(param, key, "'/' character not allowed.") end end #------------------------------------------------------------------------------# # Parameters common to all tools #------------------------------------------------------------------------------# def common_params() on('-h', '--help', HELP_DESCRIPTION) do @show_help = true end on('--version', VERSION_DESCRIPTION) do @version = true end on('--manual', MANUAL_DESCRIPTION) do @manual = true end on('--batch', BATCH_DESCRIPTION) do @batch_mode = true end on('--debug', DEBUG_DESCRIPTION) do @debug = true end end def initialize(argv, name=nil) super(argv) # Mandatory parameters. separator("") separator("MANDATORY PARAMETERS") mandatory_params() # Optional parameters. separator("") separator("OPTIONAL PARAMETERS") common_params() optional_params() # Parse the command line parameters. parse!(argv) unless early_exit? validate_params() set_defaults() end end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundleimageparameters.rb0000644000000000000000000000261312050102225022656 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/bundlemachineparameters' # The Bundle Image command line parameters. class BundleImageParameters < BundleMachineParameters IMAGE_PATH_DESCRIPTION = "The path to the file system image to bundle." PREFIX_DESCRIPTION = "The filename prefix for bundled AMI files. Defaults to image name." attr_reader :image_path, :prefix def mandatory_params() super() on('-i', '--image PATH', String, IMAGE_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--image') @image_path = path end end def optional_params() super() on('-p', '--prefix PREFIX', String, PREFIX_DESCRIPTION) do |prefix| assert_good_key(prefix, '--prefix') @prefix = prefix end end def validate_params() raise MissingMandatory.new('--image') unless @image_path super() end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/migratemanifestparameters.rb0000644000000000000000000001046012050102225023560 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/parameters_base' require 'ec2/amitools/region' require 'ec2/platform/current' #------------------------------------------------------------------------------# class MigrateManifestParameters < ParametersBase include EC2::Platform::Current::Constants MANIFEST_DESCRIPTION = "The path to the manifest file." DIRECTORY_DESCRIPTION = ["The directory containing the bundled AMI parts to upload.", "Defaults to the directory containing the manifest."] EC2_CERT_PATH_DESCRIPTION = ['The path to the EC2 X509 public key certificate bundled into the AMI.', "Defaults to the certificate of the region (usually '#{Bundling::EC2_X509_CERT}')."] KERNEL_DESCRIPTION = "Kernel id to bundle into the AMI." RAMDISK_DESCRIPTION = "Ramdisk id to bundle into the AMI." USER_DESCRIPTION = "The user's AWS access key ID." PASS_DESCRIPTION = "The user's AWS secret access key." NO_MAPPING_DESCRIPTION = "Do not perform automatic mappings." REGION_DESCRIPTION = "Region to look up in the mapping file." attr_accessor :user_pk_path, :user_cert_path, :user, :pass, :ec2_cert_path, :manifest_path, :kernel_id, :ramdisk_id, :use_mapping, :region #----------------------------------------------------------------------------# def mandatory_params() on('-c', '--cert PATH', String, USER_CERT_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--cert') @user_cert_path = path end on('-k', '--privatekey PATH', String, USER_PK_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--privatekey') @user_pk_path = path end on('-m', '--manifest PATH', String, MANIFEST_DESCRIPTION) do |manifest| assert_file_exists(manifest, '--manifest') @manifest_path = manifest end end #----------------------------------------------------------------------------# def optional_params() on('-a', '--access-key USER', String, USER_DESCRIPTION) do |user| @user = user end on('-s', '--secret-key PASSWORD', String, PASS_DESCRIPTION) do |pass| @pass = pass end on('--ec2cert PATH', String, *EC2_CERT_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--ec2cert') @ec2_cert_path = path end on('--kernel KERNEL_ID', String, KERNEL_DESCRIPTION) do |kernel_id| @kernel_id = kernel_id end on('--ramdisk RAMDISK_ID', String, RAMDISK_DESCRIPTION) do |ramdisk_id| @ramdisk_id = ramdisk_id end on('--no-mapping', String, NO_MAPPING_DESCRIPTION) do @use_mapping = false end on('--region REGION', String, REGION_DESCRIPTION) do |region| @region = region end end #----------------------------------------------------------------------------# def validate_params() raise MissingMandatory.new('--manifest') unless @manifest_path raise MissingMandatory.new('--cert') unless @user_cert_path raise MissingMandatory.new('--privatekey') unless @user_pk_path @use_mapping = true if @use_mapping.nil? # False is different. if @use_mapping raise ParameterExceptions::Error.new('If using automatic mapping, --region must be provided.') unless @region raise ParameterExceptions::Error.new('If using automatic mapping, --access-key must be provided.') unless @user raise ParameterExceptions::Error.new('If using automatic mapping, --secret-key must be provided.') unless @pass end end #----------------------------------------------------------------------------# def set_defaults() @ec2_cert_path ||= AwsRegion.is_gov(@region) ? Bundling::EC2_X509_GOV_CERT : Bundling::EC2_X509_CERT end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/xmlutil.rb0000644000000000000000000000365612050102225020024 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'rexml/document' require 'stringio' # Module containing utility XML manipulation functions. module XMLUtil # Extract the string representation of the specified XML element name # _element_name_ from the XML string _xml_data_ def XMLUtil.get_xml(xml_data, element_name) start_tag = '<'+element_name+'>' end_tag = '' return nil if (start_idx = xml_data.index(start_tag)).nil? return nil if (end_idx = xml_data.index(end_tag)).nil? end_idx += end_tag.size - 1 xml_data[start_idx..end_idx] end #----------------------------------------------------------------------------# # Trivially escape the XML string _xml_, by making the following substitutions: # * & for & # * < for < # * > for > # Return the escaped XML string. def XMLUtil::escape( xml ) escaped_xml = xml.gsub( '&', '&' ) escaped_xml.gsub!( '<', '<' ) escaped_xml.gsub!( '>', '>' ) return escaped_xml end #----------------------------------------------------------------------------# # Trivially unescape the escaped XML string _escaped_xml_, by making the # following substitutions: # * & for & # * < for < # * > for > # Return the XML string. def XMLUtil::unescape( escaped_xml ) xml = escaped_xml.gsub( '<', '<' ) xml.gsub!( '>', '>' ) xml.gsub!( '&', '&' ) return xml end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/s3toolparameters.rb0000644000000000000000000000760412050102225021632 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/parameters_base' #------------------------------------------------------------------------------# class S3ToolParameters < ParametersBase BUCKET_DESCRIPTION = ["The bucket to use. This is an S3 bucket,", "followed by an optional S3 key prefix using '/' as a delimiter."] MANIFEST_DESCRIPTION = "The path to the manifest file." DELEGATION_TOKEN_DESCRIPTION = "The delegation token pass along to the AWS request." URL_DESCRIPTION = "The S3 service URL. Defaults to https://s3.amazonaws.com." PROFILE_PATH = '/latest/meta-data/iam/security-credentials/' PROFILE_HOST = '169.254.169.254' attr_accessor :bucket, :keyprefix, :user, # This now contains all the creds. :pass, # pass is just kept for backwards compatibility. :url #------------------------------------------------------------------------------# def split_container(container) splitbits = container.sub(%r{^/*},'').sub(%r{/*$},'').split("/") bucket = splitbits.shift keyprefix = splitbits.join("/") keyprefix += "/" unless keyprefix.empty? @keyprefix = keyprefix @bucket = bucket end #----------------------------------------------------------------------------# def mandatory_params() on('-b', '--bucket BUCKET', String, *BUCKET_DESCRIPTION) do |container| @container = container split_container(@container) end on('-a', '--access-key USER', String, USER_DESCRIPTION) do |user| @user = {} if @user.nil? @user['aws_access_key_id'] = user end on('-s', '--secret-key PASSWORD', String, PASS_DESCRIPTION) do |pass| @user = {} if @user.nil? @user['aws_secret_access_key'] = pass @pass = pass end on('-t', '--delegation-token TOKEN', String, DELEGATION_TOKEN_DESCRIPTION) do |token| @user = {} if @user.nil? @user['aws_delegation_token'] = token end end #----------------------------------------------------------------------------# def optional_params() on('--url URL', String, URL_DESCRIPTION) do |url| @url = url end end #----------------------------------------------------------------------------# def get_creds_from_instance_profile end def get_creds_from_instance_profile begin require 'json' require 'net/http' profile_name = Net::HTTP.get(PROFILE_HOST, PROFILE_PATH) unless (profile_name.nil? || profile_name.strip.empty?) creds_blob = Net::HTTP.get(PROFILE_HOST, PROFILE_PATH + profile_name.strip) creds = JSON.parse(creds_blob) @user = { 'aws_access_key_id' => creds['AccessKeyId'], 'aws_secret_access_key' => creds['SecretAccessKey'], 'aws_delegation_token' => creds['Token'], } @pass = creds['SecretAccessKey'] end rescue Exception => e @user = nil end end def validate_params() unless @user get_creds_from_instance_profile end raise MissingMandatory.new('--access-key') unless @user && @user['aws_access_key_id'] raise MissingMandatory.new('--secret-key') unless @pass raise MissingMandatory.new('--bucket') unless @container end #----------------------------------------------------------------------------# def set_defaults() @url ||= 'https://s3.amazonaws.com' end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/fileutil.rb0000644000000000000000000001505512050102225020137 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # Utility class and methods. require 'ec2/amitools/exception' require 'tmpdir' require 'fileutils' require 'pathname' require 'tempfile' require 'zlib' ## # Module containing file utility methods. # module FileUtil include Zlib BUFFER_SIZE = 1024 * 1024 # Buffer size in bytes. PART_SUFFIX = '.part.' #----------------------------------------------------------------------------# ## # Assert the specified file exists. # An exception is raised if it does not. # def FileUtil.assert_exists(filename) raise FileError(filename, 'a required file could not be found') unless exists?(filename) end #----------------------------------------------------------------------------# ## # Compress the specified file and return the path to the temporary compressed # file that will be deleted upon termination of the process. # def FileUtil.compress(filename) outfilename = filename+'.gz' GzipWriter.open(outfilename) do |outfile| begin File.open(filename, 'r') do |infile| while not (infile.eof) outfile.write(infile.read(BUFFER_SIZE)) end end ensure end end outfilename end #----------------------------------------------------------------------------# ## # Expand ((|src_filename|)) to ((|dst_filename|)). # def FileUtil.expand(src_filename, dst_filename) GzipReader.open(src_filename) do |gzfile| File.open(dst_filename, 'w') do |file| while not (gzfile.eof?) file.write(gzfile.read(BUFFER_SIZE)) end end end end #----------------------------------------------------------------------------# ## # Split the specified file into chunks of the specified size. # yields the to a block which writes the actual chunks # The chunks are output to the local directory. # Typical invocation looks like: # FileUtil.split('file',5){|sf,cf,cs| ChunkWriter.write_chunk(sf,cf,cs)} # # ((|filename|)) The file to split. # ((|part_name_prefix|)) The prefix for the parts filenames. # ((|cb_size|)) The chunk size in bytes. # ((|dst_dir|)) The destination to create the file parts in. # # Returns a list of the created filenames. # def FileUtil.split(filename, part_name_prefix, cb_size, dst_dir) begin # Check file exists and is accessible. begin file = File.new(filename, File::RDONLY) rescue SystemCallError => e raise FileError.new(filename, "could not open file to split", e) end # Create the part file upfront to catch any creation/access errors # before writing out data. nr_parts = (Float(File.size(filename)) / Float(cb_size)).ceil part_names = [] nr_parts.times do |i| begin nr_parts_digits = nr_parts.to_s.length part_name_suffix = PART_SUFFIX + i.to_s.rjust(nr_parts_digits).gsub(' ', '0') part_names[i] = part_name = part_name_prefix + part_name_suffix FileUtils.touch File.join(dst_dir, part_name) rescue SystemCallError => e raise FileError.new(filename, "could not create part file", e) end end # Write parts to files. part_names.each do |part_file_name| File.open(File.join(dst_dir, part_file_name), 'w+') do |pf| write_chunk(file, pf, cb_size) end end part_names ensure file.close if not file.nil? end end #----------------------------------------------------------------------------# ## # Concatenate the specified files into a single file. # If the specified output file already exists it will be overwritten. # ((|filenames|)) An ordered collection of the names of split files. # ((|out_filename|)) The output filename. # def FileUtil.cat(filenames, out_filename) File.open(out_filename, 'w') do |of| filenames.each do |filename| File.open(filename) do |file| while not (file.eof?) of.write(file.read(BUFFER_SIZE)) of.flush end end end end end #----------------------------------------------------------------------------# def FileUtil.exists?(fullname) FileTest.exists?(fullname) end #----------------------------------------------------------------------------# def FileUtil.directory?(fullname) File::Stat.new(fullname).directory? end #----------------------------------------------------------------------------# def FileUtil.symlink?(fullname) File::Stat.new(fullname).symlink? end #----------------------------------------------------------------------------# def FileUtil.tempdir(basename, tmpdir=Dir::tmpdir, tries=10) tmpdir = '/tmp' if $SAFE > 0 and tmpdir.tainted? fail = 0 tmpname = nil begin Thread.critical = true begin tmpname = File.join(tmpdir, sprintf('%s%d.%s', basename, $$, Time.now.to_f.to_s)) end until !File.exist? tmpname rescue fail += 1 retry if fail < tries raise "failed to generate a temporary directory name '%s'" % tmpname ensure Thread.critical = false end tmpname end #----------------------------------------------------------------------------# def FileUtil.size(f) total = `du -s #{f}`.split[0].to_i rescue nil if total.nil? or $?.exitstatus != 0 total = 0 Find.find(f) do |s| total += File.directory?(s) ? 0 : File.size(s) end end total end #----------------------------------------------------------------------------# ## # Write chunk to file. # ((|sf|)) Source file. # ((|cf|)) Chunk file. # ((|cs|)) Chunk size in bytes. # Returns true if eof was encountered. # def FileUtil.write_chunk(sf, cf, cs) cb_written = 0 # Bytes written. cb_left = cs # Bytes left to write in this chunk. while (!sf.eof? && cb_left > 0) do buf = sf.read(BUFFER_SIZE < cb_left ? BUFFER_SIZE : cb_left) cf.write(buf) cb_written += buf.length cb_left = cs - cb_written end sf.eof end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/format.rb0000644000000000000000000001050012050102225017600 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'stringio' module Format #----------------------------------------------------------------------------# ## # Convert the binary data ((|data|)) to an ASCII string of hexadecimal digits # that represents it. # # E.g. if ((|data|)) is the string of bytes 0x1, 0x1A, 0xFF, it is converted # to the string "011AFF". # def Format.bin2hex(data) hex = StringIO.new data.unpack('H*').each {|digit| hex.write(digit)} hex.string end #----------------------------------------------------------------------------# ## # Breaks ((|data|)) into blocks of size ((|blocksize|)). The last block maybe # less than ((|blocksize||). # def Format.block(data, blocksize) blocks = Array.new read = 0 while read < data.size left = data.size - read blocks << data[read, (left < blocksize) ? left : blocksize] read += (left < blocksize ? left : blocksize) end blocks end #----------------------------------------------------------------------------# ## # Convert ASCII string of hexadecimal digits ((|hex|)) into the binary data it # represents. If there are an odd number of hexedecimal digits in ((|hex|)) it # is left padded with a leading '0' character. # # E.g. if ((|hex|)) is "11AFF", it is converted to the string of bytes 0x1, # 0x1A, 0xFF. # def Format.hex2bin(hex) hex = '0' + hex unless (hex.size % 2) == 0 data = StringIO.new hex.to_a.pack('H*').each {|digit| data.write(digit)} data.string end #----------------------------------------------------------------------------# ## # Return a single character string containing ((|int|)) converted to a single # byte unsigned integer. The operand must be less than 256. # def Format.int2byte int raise ArgumentError.new('argument greater than 255') unless int < 256 int.chr end #----------------------------------------------------------------------------# # Convert integer _i_ to an unsigned 16 bit int packed into two bytes in big # endian order. def Format::int2int16( i ) raise ArgumentError.new( 'argument greater than 65535' ) unless i < 65536 hi_byte = ( i >> 8 ).chr lo_byte = ( i & 0xFF).chr return [ hi_byte, lo_byte ] end #----------------------------------------------------------------------------# ## # Pad data string ((|data|)) according to the PKCS #7 padding scheme. # def Format.pad_pkcs7(data, blocksize) raise ArgumentError.new("invalid data: #{data.to_s}") unless data and data.kind_of? String raise ArgumentError.new("illegal blocksize: #{blocksize}") unless blocksize > 0x0 and blocksize < 0xFF # Determine the number of padding characters required. If the data size is # divisible by the blocksize, a block of padding is required. nr_padchars = blocksize - (data.size % blocksize) nr_padchars = blocksize unless nr_padchars != 0 # Create padding, where the padding byte is the number of padding bytes. padchar = nr_padchars.chr padding = padchar * nr_padchars data + padding end #----------------------------------------------------------------------------# ## # Pad ((|data|)) according to the PKCS #7 padding scheme. # def Format.unpad_pkcs7(data, blocksize) raise ArgumentError.new("illegal blocksize: #{blocksize}") unless blocksize > 0x0 and blocksize < 0xFF raise ArgumentError.new("invalid data: #{data.to_s}") unless data and data.kind_of? String raise ArgumentError.new("invalid data size: #{data.size}") unless data.size > 0 and (data.size % blocksize) == 0 nr_padchars = data[data.size - 1] raise ArgumentError.new("data padding character invalid: #{nr_padchars}") unless (nr_padchars > 0 and nr_padchars <= blocksize) data[0, data.size - nr_padchars] end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/uploadbundleparameters.rb0000644000000000000000000000626712050102225023071 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/s3toolparameters' require 'ec2/amitools/region' #------------------------------------------------------------------------------# class UploadBundleParameters < S3ToolParameters MANIFEST_DESCRIPTION = "The path to the manifest file." ACL_DESCRIPTION = ["The access control list policy [\"public-read\" | \"aws-exec-read\"].", "Defaults to \"aws-exec-read\"."] DIRECTORY_DESCRIPTION = ["The directory containing the bundled AMI parts to upload.", "Defaults to the directory containing the manifest."] PART_DESCRIPTION = "Upload the specified part and upload all subsequent parts." RETRY_DESCRIPTION = "Automatically retry failed uploads." SKIP_MANIFEST_DESCRIPTION = "Do not upload the manifest." LOCATION_DESCRIPTION = "The location of the bucket to upload to [#{AwsRegion.s3_locations.join(',')}]." attr_accessor :manifest, :acl, :directory, :part, :retry, :skipmanifest, :location #----------------------------------------------------------------------------# def mandatory_params() super() on('-m', '--manifest PATH', String, MANIFEST_DESCRIPTION) do |manifest| assert_file_exists(manifest, '--manifest') @manifest = manifest end end #----------------------------------------------------------------------------# def optional_params() super() on('--acl ACL', String, *ACL_DESCRIPTION) do |acl| assert_option_in(acl, ['public-read', 'aws-exec-read'], '--acl') @acl = acl end on('-d', '--directory DIRECTORY', String, *DIRECTORY_DESCRIPTION) do |directory| assert_directory_exists(directory, '--directory') @directory = directory end on('--part PART', Integer, PART_DESCRIPTION) do |part| @part = part end on('--retry', RETRY_DESCRIPTION) do @retry = true end on('--skipmanifest', SKIP_MANIFEST_DESCRIPTION) do @skipmanifest = true end on('--location LOCATION', LOCATION_DESCRIPTION) do |location| assert_option_in(location, AwsRegion.s3_locations, '--location') @location = location @location = :unconstrained if @location == "US" end end #----------------------------------------------------------------------------# def validate_params() super() raise MissingMandatory.new('--manifest') unless @manifest end #----------------------------------------------------------------------------# def set_defaults() super() @acl ||= 'aws-exec-read' @directory ||= File::dirname(@manifest) end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/xmlbuilder.rb0000644000000000000000000001644112050102225020471 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # This class makes it easy to build up xml docs, xpath style. # Usage: # # # Create an XMLBuilder: # doc = REXML::Document.new(some_xml) # builder = XMLBuilder.new(doc) # # Add some text elements # builder[/book/title] = 'Atlas Shrugged' # builder[/book/author] = 'Ann Raynd' # # Add an attribute # builder[/book/author@salutation] = 'Mrs.' # # Results in the following xml: # # Atlas Shrugged # Ann Raynd # # # Notes on Usage: # - If the path starts with a '/' the path is absolute. # - If the path does not start with a '/' the path is relative. # - When adding an element the return value is an xml builder object anchored at that path # - When adding an attrubte the return value is ... ??? dunno what is should be. nil? # - To set the element value use XMLBuilder[]=(string) or XMLBuilder[]=(XMLBuilder) or XMLBuilder[]=(node) # - To add an element do builder[path] << string or builder[path] << element or builder[path] << node # - If you assign a nil value the value will not be set and the path elements not created. require 'rexml/document' class XMLBuilder attr_reader :root # Create a new XMLBuilder rooted at the given element, or at a new document if no root is given def initialize(root = nil) @root = root || REXML::Document.new() end # Retrieve a builder for the element at the given path def [](path) nodes = PathParser.parse(path) rexml_node = nodes.inject(@root) do |rexml_node, parser_node| parser_node.walk_visit(rexml_node) end XMLBuilder.new(nodes.last.retrieve_visit(rexml_node)) end # Set the value of the element or attribute at the given path def []=(path, value) # Don't create elements or make assignments for nil values return if value.nil? nodes = PathParser.parse(path) rexml_node = nodes.inject(@root) do |rexml_node, parser_node| parser_node.walk_visit(rexml_node) end nodes.last.assign_visit(rexml_node, value) end # Parses an xpath like expression class PathParser def PathParser.parse(path) nodes = path.split('/') @nodes = [] first = true while (nodes.length > 0) node = Node.new(first, nodes) first = false @nodes << Document.new() if node.document @nodes << Element.new(node.element, node.index) if node.element @nodes << Attribute.new(node.attribute) if node.attribute end @nodes end # Helper class used by PathParser class Node attr_reader :document attr_reader :element attr_reader :index attr_reader :attribute # Regex for parsing path if the form element[index]@attribute # where [index] and @attribute are optional @@attribute_regex = /^@(\w+)$/ @@node_regex = /^(\w+)(?:\[(\d+)\])?(?:@(\w+))?$/ # Nodes is path.split('/') def initialize(allow_document, nodes) if allow_document && nodes[0] == '' @document = true nodes.shift return end nodes.shift while nodes[0] == '' node = nodes.shift if (match = @@node_regex.match(node)) @element = match[1] @index = match[2].to_i || 0 elsif (match = @@attribute_regex.match(node)) @attribute = match[1] else raise 'Invalid path: Node must be of the form element[index] or @attribute' if document end end end # Node classes # Each class represents a node type. An array of these classes is built up when # the path is parsed. Each node is then visited by the current rexml node and # (depending on which visit function is called) the action taken. # # The visit function are: # - walk: Walks down the xml dom # - assign: Assigns the value to rexml node # - retrieve: Retrives the value of the rexml node # # Different node types implement different behaviour types. For example retrieve is # illegal on Attribute nodes, but on Document and Attribute nodes the given node is returned. class Document def initialize() end # Move to the document node (top of the xml dom) def walk_visit(rexml_node) return rexml_node if rexml_node.is_a?(REXML::Document) raise "No document set on node. #{rexml_node.name}" if rexml_node.document == nil rexml_node.document end def assign_visit(document_node, value) raise 'Can only assign REXML::Elements to document nodes' if !value.is_a?(REXML::Element) raise 'Expected a document node' if !document_node.is_a?(REXML::Element) if doc.root doc.replace_child(doc.root, value) else doc.add_element(element) end end def retrieve_visit(document_node) document_node end end class Element attr_reader :name attr_reader :index def initialize(name, index) @name = name @index = index end # Move one element down in the dom def walk_visit(rexml_node) elements = rexml_node.get_elements(name) raise "Invalid index #{index} for element #{@name}" if @index > elements.length # Create a node if it doesn't exist if @index == elements.length new_element = REXML::Element.new(@name) rexml_node.add_element(new_element) new_element else elements[@index] end end def assign_visit(rexml_node, value) if value.is_a?(String) rexml_node.text = value elsif value.is_a?(REXML::Element) value.name = rexml_node.name raise "Node #{rexml_node.name} does not have a parent node" if rexml_node.parent.nil? rexml_node.parent.replace_child(rexml_node, value) else raise 'Can only assign a String or a REXML::Element to an element' end end def retrieve_visit(rexml_node) rexml_node end end class Attribute attr_reader :name def initialize(name) @name = name end # Stays on the same node in the dom def walk_visit(rexml_node) rexml_node end def assign_visit(rexml_node, value) raise 'Can only assign an attribute to an element.' if !rexml_node.is_a?(REXML::Element) rexml_node.attributes[@name] = value.to_s end def retrieve_visit(rexml_node) raise 'Accessor not valid for paths with an attribute' end end end end # Test code if $0 == __FILE__ def assert_true(expr) raise 'expected true' if !expr end doc = REXML::Document.new() builder = XMLBuilder.new(doc) root_builder = builder['/root'] root_builder['nested'] = 'more text' root_builder['/root/bnode'] = 'name' root_builder['/root/bnode/@attr'] = 'attr' puts doc end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/region.rb0000644000000000000000000000313312050102225017577 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. # Module to hold region informations # module AwsRegion AWS_REGIONS = [ 'eu-west-1', 'us-east-1', 'us-gov-west-1', 'us-west-1', 'us-west-2', 'ap-southeast-1', 'ap-southeast-2', 'ap-northeast-1', 'sa-east-1', ] S3_LOCATIONS = [ 'EU', 'US', 'us-gov-west-1', 'us-west-1', 'us-west-2', 'ap-southeast-1', 'ap-southeast-2', 'ap-northeast-1', 'sa-east-1', ] GOV_REGIONS = [ 'us-gov-west-1', ] module_function def guess_region_from_s3_bucket(location) if (location == "EU") return "eu-west-1" elsif ((location == "US") || (location == "") || (location.nil?)) return "us-east-1" else return location end end def get_s3_location(region) if (region == "eu-west-1") return 'EU' elsif (region == "us-east-1") return :unconstrained else return region end end def is_gov(region) return GOV_REGIONS.include?(region) end def regions AWS_REGIONS end def s3_locations S3_LOCATIONS end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/unbundleparameters.rb0000644000000000000000000000446312050102225022223 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/parameters_base' class UnbundleParameters < ParametersBase MANIFEST_DESCRIPTION = "The path to the AMI manifest file." SOURCE_DESCRIPTION = 'The directory containing bundled AMI parts to unbundle. Defaults to ".".' DESTINATION_DESCRIPTION = 'The directory to unbundle the AMI into. Defaults to the ".".' attr_accessor :manifest_path, :user_pk_path, :source, :destination #----------------------------------------------------------------------------# def mandatory_params() on('-k', '--privatekey PATH', String, USER_PK_PATH_DESCRIPTION) do |path| assert_file_exists(path, '--privatekey') @user_pk_path = path end on('-m', '--manifest PATH', String, MANIFEST_DESCRIPTION) do |manifest| assert_file_exists(manifest, '--manifest') @manifest_path = manifest end end #----------------------------------------------------------------------------# def optional_params() on('-s', '--source DIRECTORY', String, SOURCE_DESCRIPTION) do |directory| assert_directory_exists(directory, '--source') @source = directory end on('-d', '--destination DIRECTORY', String, DESTINATION_DESCRIPTION) do |directory| assert_directory_exists(directory, '--destination') @destination = directory end end #----------------------------------------------------------------------------# def validate_params() raise MissingMandatory.new('--manifest') unless @manifest_path raise MissingMandatory.new('--privatekey') unless @user_pk_path end #----------------------------------------------------------------------------# def set_defaults() @source ||= Dir::pwd() @destination ||= Dir::pwd() end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/util.rb0000644000000000000000000003467112050102225017304 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'monitor' require 'thread' require 'syslog' ## # generate a unique identifier used for filenames # def gen_ident() (0..19).inject("") {|ident, n| ident+(?A + Kernel.rand(26)).chr} end #------------------------------------------------------------------------------# # A thread local buffer manager. Provide's each thread with a single # pre-allocated IO buffer. IMPORTANT: as these buffers are indexed in a # hashtable they will not be freed until the application closes. If a thread # needs to free the memory its buffer is using, it must call # delete_buffer and ensure it has no references to the buffer. class ThreadLocalBuffer POWER = 12 # Log SIZE base 2 SIZE = 2**POWER # Size of the buffer. @@buffers = {} @@buffers.extend( MonitorMixin ) #----------------------------------------------------------------------------# # Return the thread's buffer. def ThreadLocalBuffer.get_buffer @@buffers.synchronize do @@buffers[Thread.current] = new_buffer unless @@buffers.has_key?( Thread.current ) @@buffers[Thread.current] end end #----------------------------------------------------------------------------# # Delete the threads buffer. def delete_buffer buffer @@buffers.delete buffer end #----------------------------------------------------------------------------# def ThreadLocalBuffer.new_buffer buffer = String.new buffer = Format::hex2bin( '00' ) POWER.times { buffer << buffer } buffer end private_class_method :new_buffer end #------------------------------------------------------------------------------# ## # Base class for XML-RPC structures. Stores key and values pairs. Key names are # mapped to method names by converting '-' to '_' characters. # class XMLRPCStruct < Hash # _members_ A list of the structure's key names or nil if any key names # are allowed. def initialize members unless members.kind_of? Array or members.nil? raise ArgumentError.new( "invalid members argument" ) end @members = members end # Provide direct access to individual instance elements by methods named after # the element's key. def method_missing( method_symbol, argument=nil ) # Here kid, play with this loaded gun... method = method_symbol.to_s # Determine if setter or getter call and remove '=' if setter. setter = /[\S]+=/.match(method) member = (setter ? method.slice(0, method.size - 1) : method) # Map method name to member name. member = member.gsub('_', '-') # If valid attribute set or get accordingly. If the member list is nil then # any members are allowed. if @members.nil? or @members.include?( member ) if setter raise ArgumentError, "value for key #{member} may not be nil" if argument.nil? self[member] = argument else self[member] end else raise NoMethodError.new( method ) end end end #------------------------------------------------------------------------------# ## # Note to self - use log4r next time ;) # class Log #----------------------------------------------------------------------------# # @deprecated use Priority instead class Verbosity private_class_method :new @@levels = Hash.new def initialize value @value = value @@levels[value] = self end V0 = new 0 # Unhandled exceptions only. V1 = new 1 # As for 0 but with error messages. V2 = new 2 # As for 1 but with informational messages. V3 = new 3 # As for 2 but with XML-RPC logging. V4 = new 4 # As for 3 but with Xen logging. V5 = new 5 # As for 4 but with debugging messages. attr_accessor :value def >= operand @value >= operand.value end def Verbosity.from_string s level = s.to_i if not @@levels[level] raise ArgumentError.new("invalid logging verbosity level #{level}") else @@levels[level] end end def to_priority case self when V0 return Priority::ALERT when V1 return Priority::ERR when V2 return Priority::WARNING when V3 return Priority::NOTICE when V4 return Priority::INFO else return Priority::DEBUG end end end #----------------------------------------------------------------------------# class Facility private_class_method :new def initialize(name) @name = name @value = (name == "AES")?(12<<3):eval("Syslog::LOG_#{name}") end AUTHPRIV = new "AUTHPRIV" CRON = new "CRON" DAEMON = new "DAEMON" FTP = new "FTP" KERN = new "KERN" LOCAL0 = new "LOCAL0" LOCAL1 = new "LOCAL1" LOCAL2 = new "LOCAL2" LOCAL3 = new "LOCAL3" LOCAL4 = new "LOCAL4" LOCAL5 = new "LOCAL5" LOCAL6 = new "LOCAL6" LOCAL7 = new "LOCAL7" LPR = new "LPR" MAIL = new "MAIL" NEWS = new "NEWS" SYSLOG = new "SYSLOG" USER = new "USER" UUCP = new "UUCP" AES = new "AES" attr_accessor :value attr_accessor :name def to_s "Facility[LOG_#{@name}]" end end #----------------------------------------------------------------------------# class Priority include Comparable private_class_method :new def initialize(name) @name = name @value = eval("Syslog::LOG_#{name}") end EMERG = new "EMERG" # 0 ALERT = new "ALERT" # 1 CRIT = new "CRIT" # 2 ERR = new "ERR" # 3 WARNING = new "WARNING" # 4 NOTICE = new "NOTICE" # 5 INFO = new "INFO" # 6 DEBUG = new "DEBUG" # 7 attr_accessor :value attr_accessor :name def <=>(priority) @value <=> priority.value end def to_s "Priority[LOG_#{@name}]" end end #----------------------------------------------------------------------------# @@facility = Facility::AES @@priority = Priority::INFO @@identity = nil @@streams_mutex = Mutex.new @@streams = [] #----------------------------------------------------------------------------# ## # Set the verbosity of the logging. # @deprecated use set_priority def Log.set_verbosity(verbosity) set_priority(verbosity.to_priority) end #----------------------------------------------------------------------------# ## # Set the IO instance to log to. # @deprecated use add_stream def Log.set_io(io) add_stream(io) end #----------------------------------------------------------------------------# ## # Log a debug message. def Log.debug(msg=nil) if block_given? write(Priority::DEBUG) {yield} else write(Priority::DEBUG) {msg} end end #----------------------------------------------------------------------------# ## # Log a warning message. def Log.warn(msg=nil) if block_given? write(Priority::WARNING) {yield} else write(Priority::WARNING) {msg} end end #----------------------------------------------------------------------------# ## # Log an informational message. def Log.info(msg=nil) if block_given? write(Priority::INFO) {yield} else write(Priority::INFO) {msg} end end #----------------------------------------------------------------------------# ## # Log an error message. def Log.err(msg=nil) if block_given? write(Priority::ERR) {yield} else write(Priority::ERR) {msg} end end #----------------------------------------------------------------------------# ## # Log a warning message. def Log.warn(msg=nil) if block_given? write(Priority::WARNING) {yield} else write(Priority::WARNING) {msg} end end #----------------------------------------------------------------------------# ## # Log an unhandled exception. # @deprecated use write def Log.exception(e) if block_given? write(Priority::ALERT) {yield} else write(Priority::ALERT) {Log.exception_str(e)} end end #----------------------------------------------------------------------------# ## # Log an informational message. # @deprecated use write def Log.msg(msg) write(Verbosity::V2.to_priority) {msg} end #----------------------------------------------------------------------------# # @deprecated use write def Log.xen_msg(msg) write(Verbosity::V4.to_priority) {msg} end #----------------------------------------------------------------------------# # @deprecated use write def Log.xmlrpcmethod_call(name, *paramstructs) write(Verbosity::V3.to_priority) {Log.xmlrpcmethod_call_str(name, paramstructs)} end #----------------------------------------------------------------------------# # @deprecated use write def Log.xmlrpcmethod_return(name, value) write(Verbosity::V3.to_priority) {Log.xmlrpcmethod_return_str(name, value)} end #----------------------------------------------------------------------------# # @deprecated use write def Log.xmlrpcfault(xmlrpc_method, fault) write(Verbosity::V3.to_priority) {Log.xmlrpcfault_str(xmlrpc_method, fault)} end #----------------------------------------------------------------------------# ## # Add an additional stream (like a file, or $stdout) to send # log output to. # def Log.add_stream(stream) @@streams_mutex.synchronize do @@streams.push(stream) @@streams.delete_if { |io| io.closed? } end end #----------------------------------------------------------------------------# def Log.exception_str(e) e.message + "\n" + e.backtrace.to_s end #----------------------------------------------------------------------------# def Log.xmlrpcfault_str(xmlrpc_method, fault) "XML-RPC method fault\nmethod: #{xmlrpc_method}\nfault code: #{fault.faultCode}\nfault string: #{fault.faultString}" end #----------------------------------------------------------------------------# def Log.xmlrpcmethod_call_str(name, *paramstructs) msg = "name: #{name}\n" paramstructs.each_index { |i| msg += "parameter #{i + 1}: #{paramstructs[i].inspect}\n" } "XML-RPC method call\n#{msg}" end #----------------------------------------------------------------------------# def Log.xmlrpcmethod_return_str(name, value) msg = "name: #{name}\nvalue: #{value.inspect}" "XML-RPC method return\n#{msg}" end #----------------------------------------------------------------------------# ## # Set the minimum priority of the logging. Messages logged with a # lower (less urgent) priority will be ignored. # def Log.set_priority(priority) @@priority = priority end #----------------------------------------------------------------------------# ## # Set the facility to log messages against when no explicit facility is # provided. # def Log.set_facility(facility) @@facility = facility end #----------------------------------------------------------------------------# ## # Set the identity to log messages against when no explicit identity is # provided. If no identity is provided (either using this method or explicitly # when logging) the system will use the application name as the identity. # def Log.set_identity(identity) @@identity = identity end #----------------------------------------------------------------------------# SYSLOG_OPTS = (Syslog::LOG_PID | Syslog::LOG_CONS) #----------------------------------------------------------------------------# def Log.time Time.new.to_s end private_class_method :time #----------------------------------------------------------------------------# def Log.write(priority=Priority::DEBUG, facility=nil, identity=nil) # If the priority of this message is below the defined priority # for logging then we don't want to do this at all. NOTE: Priorities # for syslog are defined in ascending order (so lower priorities # are more urgent). return unless priority <= @@priority return unless block_given? begin facility = (facility == nil)?(@@facility):(facility) fac_int = facility.value ident = (identity == nil)?(@@identity):(identity) msg = yield Syslog.open(ident, SYSLOG_OPTS, fac_int) do |log| log.log(priority.value, '%s', msg) end # Now pass the message onto each registered stream # Access to our list of streams is synchronized so that it can be changed # at runtime. @@streams_mutex.synchronize do @@streams.each do |stream| begin stream.puts "#{time}: #{ident}: #{priority.value}: #{msg}" stream.flush rescue Exception => e $stderr.puts 'error writing to stream [#{stream}], logging to stdout' end end end rescue Exception => e $stderr.puts "error loggin to syslog, logging to stdout: #{e}" if block_given? begin $stdout.puts "Msg: #{msg}" rescue Exception => e $stderr.puts "Block raised error: #{e}" end end end end end #------------------------------------------------------------------------------# ## # Utilities used for logging (in order for compatability between AESLogger and the previous Log class) class LogUtils ## # Prevent instantiation private_class_method :new def LogUtils.exception_str(e) e.message + "\n" + e.backtrace.to_s end def LogUtils.xmlrpcfault_str(xmlrpc_method, fault) "XML-RPC method fault : method: #{xmlrpc_method} : fault code: #{fault.faultCode} : fault string: #{fault.faultString}" end def LogUtils.xmlrpcmethod_call_str(name, *paramstructs) msg = "name: #{name} : " paramstructs.each_index { |i| msg += "parameter #{i + 1}: #{paramstructs[i].inspect} : " } "XML-RPC method call\n#{msg}" end def LogUtils.xmlrpcmethod_return_str(name, value) msg = "name: #{name} : value: #{value.inspect}" "XML-RPC method return : #{msg}" end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundle.rb0000644000000000000000000002405412050102225017572 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'open-uri' require 'ec2/amitools/fileutil' require 'ec2/amitools/manifestv20071010' require 'ec2/amitools/instance-data' require 'ec2/amitools/version' require 'ec2/platform/current' # Module containing utility methods for bundling an AMI. module Bundle class ImageType < String MACHINE = new 'machine' KERNEL = new 'kernel' RAMDISK = new 'ramdisk' VOLUME = new 'machine' def ==(o) self.object_id == o.object_id end end CHUNK_SIZE = 10 * 1024 * 1024 # 10 MB in bytes. def self.bundle_image( image_file, user, arch, image_type, destination, user_private_key_path, user_cert_path, ec2_cert_path, prefix, optional_args, debug = false, inherit = true ) begin raise "invalid image-type #{image_type}" unless image_type.is_a? Bundle::ImageType # Create named pipes. digest_pipe = File::join('/tmp', "ec2-bundle-image-digest-pipe-#{$$}") File::delete(digest_pipe) if File::exist?(digest_pipe) unless system( "mkfifo #{digest_pipe}" ) raise "Error creating named pipe #{digest_pipe}" end # If the prefix differs from the file name create a symlink # so that the file is tarred with the prefix name. if prefix and File::basename( image_file ) != prefix image_file_link = File::join( destination, prefix ) begin FileUtils.ln_s(image_file, image_file_link) rescue Exception => e raise "Error creating symlink to image file, #{e.message}." end image_file = image_file_link end # Load and generate necessary keys. name = prefix || File::basename( image_file ) manifest_file = File.join( destination, name + '.manifest.xml') bundled_file_path = File::join( destination, name + '.tar.gz.enc' ) user_public_key = Crypto::certfile2pubkey( user_cert_path ) ec2_public_key = Crypto::certfile2pubkey( ec2_cert_path ) key = Format::bin2hex( Crypto::gensymkey ) iv = Format::bin2hex( Crypto::gensymkey ) # Bundle the AMI. # The image file is tarred - to maintain sparseness, gzipped for # compression and then encrypted with AES in CBC mode for # confidentiality. # To minimize disk I/O the file is read from disk once and # piped via several processes. The tee is used to allow a # digest of the file to be calculated without having to re-read # it from disk. tar = EC2::Platform::Current::Tar::Command.new.create.dereference.sparse tar.owner(0).group(0) tar.add(File::basename( image_file ), File::dirname( image_file )) openssl = EC2::Platform::Current::Constants::Utility::OPENSSL pipeline = EC2::Platform::Current::Pipeline.new('image-bundle-pipeline', debug) pipeline.concat([ ['tar', "#{openssl} sha1 < #{digest_pipe} & " + tar.expand], ['tee', "tee #{digest_pipe}"], ['gzip', 'gzip'], ['encrypt', "#{openssl} enc -e -aes-128-cbc -K #{key} -iv #{iv} > #{bundled_file_path}"] ]) digest = nil begin digest = pipeline.execute.split(/\s+/).last.strip rescue EC2::Platform::Current::Pipeline::ExecutionError => e $stderr.puts e.message exit 1 end # Split the bundled AMI. # Splitting is not done as part of the compress, encrypt digest # stream, so that the filenames of the parts can be easily # tracked. The alternative is to create a dedicated output # directory, but this leaves the user less choice. parts = Bundle::split( bundled_file_path, name, destination ) # Sum the parts file sizes to get the encrypted file size. bundled_size = 0 parts.each do |part| bundled_size += File.size( File.join( destination, part ) ) end # Encrypt key and iv. padding = OpenSSL::PKey::RSA::PKCS1_PADDING user_encrypted_key = user_public_key.public_encrypt( key, padding ) ec2_encrypted_key = ec2_public_key.public_encrypt( key, padding ) user_encrypted_iv = user_public_key.public_encrypt( iv, padding ) ec2_encrypted_iv = ec2_public_key.public_encrypt( iv, padding ) # Digest parts. part_digest_list = Bundle::digest_parts( parts, destination ) # Launch-customization data patch_in_instance_meta_data(image_type, optional_args) if inherit # Sanity-check block-device-mappings bdm = optional_args[:block_device_mapping] if bdm.is_a? Hash [ 'root', 'ami' ].each do |item| if bdm[item].to_s.strip.empty? $stdout.puts "Block-device-mapping has no '#{item}' entry. A launch-time default will be used." end end end # Create bundle manifest. $stdout.puts 'Creating bundle manifest...' manifest = ManifestV20071010.new() manifest.init(optional_args.merge({:name => name, :user => user, :image_type => image_type.to_s, :arch => arch, :reserved => nil, :parts => part_digest_list, :size => File::size( image_file ), :bundled_size => bundled_size, :user_encrypted_key => Format::bin2hex( user_encrypted_key ), :ec2_encrypted_key => Format::bin2hex( ec2_encrypted_key ), :cipher_algorithm => Crypto::SYM_ALG, :user_encrypted_iv => Format::bin2hex( user_encrypted_iv ), :ec2_encrypted_iv => Format::bin2hex( ec2_encrypted_iv ), :digest => digest, :digest_algorithm => Crypto::DIGEST_ALG, :privkey_filename => user_private_key_path, :kernel_id => optional_args[:kernel_id], :ramdisk_id => optional_args[:ramdisk_id], :product_codes => optional_args[:product_codes], :ancestor_ami_ids => optional_args[:ancestor_ami_ids], :block_device_mapping => optional_args[:block_device_mapping], :bundler_name => EC2Version::PKG_NAME, :bundler_version => EC2Version::PKG_VERSION, :bundler_release => EC2Version::PKG_RELEASE})) # Write out the manifest file. File.open( manifest_file, 'w' ) { |f| f.write( manifest.to_s ) } ensure # Clean up. if bundled_file_path and File.exist?( bundled_file_path ) File.delete( bundled_file_path ) end File::delete( digest_pipe ) if digest_pipe and File::exist?(digest_pipe) if image_file_link and File::exist?( image_file_link ) File::delete( image_file_link ) end end end def self.patch_in_instance_meta_data(image_type, optional_args) if (image_type == ImageType::VOLUME || image_type == ImageType::MACHINE ) instance_data = EC2::InstanceData.new if !instance_data.instance_data_accessible raise "Error accessing instance data. If you are not bundling on an EC2 instance use --no-inherit." else [ [:ancestor_ami_ids, instance_data.ancestor_ami_ids, Proc.new do |key, value| if (optional_args[key].nil?) ancestry = nil if value.nil? or value.to_s.empty? ancestry = [] elsif value.is_a? Array ancestry = value else ancestry = [value] end ami_id = instance_data.ami_id $stdout.puts "Unable to read instance meta-data for ami-id" if ami_id.nil? ancestry << ami_id unless(ami_id.nil? or ancestry.include?(ami_id)) optional_args[key] = ancestry if ancestry && ancestry.length > 0 end end], [:kernel_id, instance_data.kernel_id, nil], [:ramdisk_id, instance_data.ramdisk_id, nil], [:product_codes, instance_data.product_codes, nil], [:block_device_mapping, instance_data.block_device_mapping, nil], ].each do |key, value, block| begin if value.nil? $stdout.puts "Unable to read instance meta-data for #{key.to_s.gsub('_','-')}" block.call(key, value) if block else if block block.call(key, value) else optional_args[key] ||= value end end rescue $stdout.puts "Unable to set #{key.to_s.gsub('_','-')} from instance meta-data" end end end end end def self.split( filename, prefix, destination ) $stdout.puts "Splitting #{filename}..." part_filenames = FileUtil::split(filename, prefix, CHUNK_SIZE, destination) part_filenames.each { |name| puts "Created #{name}" } part_filenames end def self.digest_parts( basenames, dir ) $stdout.puts 'Generating digests for each part...' parts_digests = Array.new basenames.each do |basename| File.open(File.join(dir, basename)) do |f| parts_digests << [basename, Crypto.digest( f )] end end $stdout.puts 'Digests generated.' parts_digests end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/mapids.rb0000644000000000000000000001254112050102225017574 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/minimalec2' class KernelMappings ENDPOINT = "http://ec2.amazonaws.com" class MappingError < StandardError; end attr_reader :source_image_info attr_reader :target_image_info #----------------------------------------------------------------------------# def get_ec2_client(akid, secretkey, endpoint=nil) EC2::EC2Client.new(akid, secretkey, endpoint) end #----------------------------------------------------------------------------# def add_endpoint_from_env(endpoints) str = ENV['AMITOOLS_EC2_REGION_ENDPOINT'] if not str return end components = str.split('@').collect{|s| s.strip()} if components.size != 2 return end endpoints[components[0]] = components[1] end def get_endpoints(akid, secretkey, endpoint=nil) endpoint ||= ENDPOINT endpoints = {} ec2conn = get_ec2_client(akid, secretkey, endpoint) resp_doc = ec2conn.describe_regions() REXML::XPath.each(resp_doc.root, '/DescribeRegionsResponse/regionInfo/item') do |region| region_name = REXML::XPath.first(region, 'regionName').text region_url = REXML::XPath.first(region, 'regionEndpoint').text region_url = 'https://'+region_url unless region_url =~ %r{^https?://} endpoints[region_name] = region_url end add_endpoint_from_env(endpoints) endpoints end #----------------------------------------------------------------------------# def create_ec2_connections(akid, secretkey, endpoint=nil) @ec2conn = {} @endpoints = get_endpoints(akid, secretkey, endpoint) @endpoints.each do |region, endpoint| @ec2conn[region] = get_ec2_client(akid, secretkey, endpoint) end end #----------------------------------------------------------------------------# def get_source_images_info() @ec2conn.each do |region, connection| $stderr.puts "Getting source data from #{region} #{@source_images.inspect}..." if @verbose resp_doc = connection.describe_images(@source_images) @source_images_info = parse_imageset(resp_doc) $stderr.puts "Found #{@source_images_info.size} images in #{region}." if @verbose # We assume all the images we're trying to map are in the same region. return if @source_images_info.size > 0 end end #----------------------------------------------------------------------------# def get_target_candidates() owners = @source_images_info.map { |ii| ii['imageOwnerId'] } $stderr.puts "Getting target data from #{region} #{owners.inspect}..." if @verbose resp_doc = @ec2conn[@target_region].describe_images([], :ownerIds => owners) @target_images_info = parse_imageset(resp_doc) $stderr.puts "Found #{@target_images_info.size} images in #{region}." if @verbose end #----------------------------------------------------------------------------# def parse_imageset(resp_doc) images_info = [] resp_doc.each_element('DescribeImagesResponse/imagesSet/item') do |image| image_data = {} image.each_element() { |elem| image_data[elem.name] = elem.text } images_info << image_data end images_info end #----------------------------------------------------------------------------# def initialize(akid, secretkey, source_images, target_region, endpoint=nil) @target_region = target_region @source_images = source_images create_ec2_connections(akid, secretkey, endpoint) unless @ec2conn.has_key?(target_region) raise MappingError.new("Invalid region: #{target_region}") end end #----------------------------------------------------------------------------# def matchable_image(image) summary = {} ['imageOwnerId', 'imageType'].each { |key| summary[key] = image[key] } summary[:s3key] = image['imageLocation'].sub(%r{^/*[^/]+/},'') summary end #----------------------------------------------------------------------------# def find_missing_targets(targets) @target_images_info.each { |ii| targets.delete(ii['imageId']) } unless @target_images_info.nil? return nil if targets.empty? lookups = parse_imageset(@ec2conn[@target_region].describe_images(targets)) lookups.each { |ii| targets.delete(ii['imageId']) } return nil if targets.empty? targets end #----------------------------------------------------------------------------# def [](identifier) if @target_images_info.nil? get_source_images_info() get_target_candidates() end source_info = @source_images_info.find { |ii| ii['imageId'] == identifier } raise MappingError.new("'#{identifier}' not found.") if source_info.nil? source_match = matchable_image(source_info) target_info = @target_images_info.find { |ii| source_match == matchable_image(ii) } raise MappingError.new("Mapping for '#{identifier}' not found.") if target_info.nil? target_info['imageId'] end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/bundle_base.rb0000644000000000000000000000342312050102225020561 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/tool_base' require 'ec2/amitools/bundleparameters' class BundleTool < AMITool def user_override(name, value) if interactive? instr = interactive_prompt("Please specify a value for #{name} [#{value}]: ", name) return instr.strip unless instr.nil? or instr.strip.empty? end value end def notify(msg) $stdout.puts msg if interactive? print "Hit enter to continue anyway or Control-C to quit." gets end end def get_parameters(params_class) params = super(params_class) if params.arch.nil? params.arch = SysChecks::get_system_arch() raise "missing or bad uname" if params.arch.nil? params.arch = user_override("arch", params.arch) end unless BundleParameters::SUPPORTED_ARCHITECTURES.include?(params.arch) unless warn_confirm("Unsupported architecture [#{params.arch}].") raise EC2StopExecution.new() end end tarcheck = SysChecks::good_tar_version? raise "missing or bad tar" if tarcheck.nil? unless tarcheck unless warn_confirm("Possibly broken tar version found. Please use tar version 1.15 or later.") raise EC2StopExecution.new() end end params end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/crypto.rb0000644000000000000000000003217512050102225017644 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/format' require 'digest/sha1' require 'openssl' require 'stringio' ### # Cryptographic utilities module. # module Crypto BUFFER_SIZE = 1024 * 1024 ASYM_ALG = 'RSA' SYM_ALG = 'AES-128-CBC' DIGEST_ALG = 'SHA1' PADDING = OpenSSL::PKey::RSA::PKCS1_PADDING VERSION1 = 1 VERSION2 = 2 SHA1_FINGERPRINT_REGEX = /([a-f0-9]{2}(:[a-f0-9]{2}){15})/i #----------------------------------------------------------------------------# ## # Decrypt the specified cipher text according to the AMI Manifest Encryption # Scheme Version 1 or 2. # # ((|cipher_text|)) The cipher text to decrypt. # ((|keyio_or_keyfilename|)) The key data IO stream or the name of the private # key file. # def Crypto.decryptasym(cipher_text, keyfilename) raise ArgumentError.new('cipher_text') unless cipher_text raise ArgumentError.new('keyfilename') unless keyfilename and FileTest.exists? keyfilename # Load key. privkey = File.open(keyfilename, 'r') { |f| OpenSSL::PKey::RSA.new(f) } # Get version. version = cipher_text[0] if version == VERSION2 return Crypto.decryptasym_v2( cipher_text, keyfilename ) end raise ArgumentError.new("invalid encryption scheme versionb: #{version}") unless version == 1 # Decrypt and extract encrypted symmetric key and initialization vector. symkey_cryptogram_len = cipher_text.slice(1, 2).unpack('C')[0] symkey_cryptogram = privkey.private_decrypt( cipher_text.slice(2, symkey_cryptogram_len), PADDING) symkey = symkey_cryptogram.slice(0, 16) iv = symkey_cryptogram.slice(16, 16) # Decrypt data with the symmetric key. cryptogram = cipher_text.slice(2 + symkey_cryptogram_len..cipher_text.size) decryptsym(cryptogram, symkey, iv) end #----------------------------------------------------------------------------# ## # Decrypt the specified cipher text according to the AMI Manifest Encryption # Scheme Version 2. # # ((|cipher_text|)) The cipher text to decrypt. # ((|keyio_or_keyfilename|)) The key data IO stream or the name of the private # key file. # def Crypto.decryptasym_v2(cipher_text, keyfilename) raise ArgumentError.new('cipher_text') unless cipher_text raise ArgumentError.new('keyfilename') unless keyfilename and FileTest.exists? keyfilename # Load key. privkey = File.open(keyfilename, 'r') { |f| OpenSSL::PKey::RSA.new(f) } # Get version. version = cipher_text[0] raise ArgumentError.new("invalid encryption scheme versionb: #{version}") unless version == VERSION2 # Decrypt and extract encrypted symmetric key and initialization vector. hi_byte, lo_byte = cipher_text.slice(1, 3).unpack('CC') symkey_cryptogram_len = ( hi_byte << 8 ) | lo_byte symkey_cryptogram = privkey.private_decrypt( cipher_text.slice(3, symkey_cryptogram_len), PADDING) symkey = symkey_cryptogram.slice(0, 16) iv = symkey_cryptogram.slice(16, 16) # Decrypt data with the symmetric key. cryptogram = cipher_text.slice( ( 3 + symkey_cryptogram_len )..cipher_text.size) decryptsym(cryptogram, symkey, iv) end #----------------------------------------------------------------------------# ## # Asymmetrically encrypt the specified data using the AMI Manifest Encryption # Scheme Version 2. # # The data is encrypted with an ephemeral symmetric key and initialization # vector. The symmetric key and initialization vector are encrypted with the # specified public key and preprended to the data. # # ((|data|)) The data to encrypt. # ((|pubkey|)) The public key. # def Crypto.encryptasym(data, pubkey) raise ArgumentError.new('data') unless data raise ArgumentError.new('pubkey') unless pubkey symkey = gensymkey iv = geniv symkey_cryptogram = pubkey.public_encrypt( symkey + iv, PADDING ) data_cryptogram = encryptsym(data, symkey, iv) hi_byte, lo_byte = Format.int2int16(symkey_cryptogram.size) Format::int2byte(VERSION2) + hi_byte + lo_byte + symkey_cryptogram + data_cryptogram end #----------------------------------------------------------------------------# ## # Verify the authenticity of the data from the IO stream or string ((|data|)) # using the signature ((|sig|)) and the public key ((|pubkey|)). # # Return true iff the signature is valid. # def Crypto.authenticate(data, sig, pubkey) raise ArgumentError.new("Invalid parameter data") if data.nil? raise ArgumentError.new("Invalid parameter sig") if sig.nil? or sig.length==0 raise ArgumentError.new("Invalid parameter pubkey") if pubkey.nil? # Create IO stream if necessary. io = (data.instance_of?(StringIO) ? data : StringIO.new(data)) sha = OpenSSL::Digest::SHA1.new res = false while not (io.eof?) res = pubkey.verify(sha, sig, io.read(BUFFER_SIZE)) end res end #----------------------------------------------------------------------------# ## # Decrypt the specified cipher text file to create the specified plain text # file. # # The symmetric cipher is AES in CBC mode. 128 bit keys are used. If the plain # text file already exists it will be overwritten. # # ((|src|)) The name of the cipher text file to decrypt. # ((|dst|)) The name of the plain text file to create. # ((|key|)) The 128 bit (16 byte) symmetric key. # ((|iv|)) The 128 bit (16 byte) initialization vector. # def Crypto.decryptfile(src, dst, key, iv) raise ArgumentError.new("invalid file name: #{src}") unless FileTest.exists?(src) raise ArgumentError.new("invalid key") unless key and key.size == 16 raise ArgumentError.new("invalid iv") unless iv and iv.size == 16 pio = IO.popen( "openssl enc -d -aes-128-cbc -in #{src} -out #{dst} -K #{Format::bin2hex(key)} -iv #{Format::bin2hex(iv)} 2>&1" ) result = pio.read pio.close raise "error decrypting file #{src}: #{result}" if result.strip != '' end #----------------------------------------------------------------------------# ## # Decrypt _ciphertext_ using _key_ and _iv_ using AES-128-CBC. # def Crypto.decryptsym(plaintext, key, iv) raise ArgumentError.new("plaintext must be a String") unless plaintext.is_a? String raise ArgumentError.new("invalid key") unless key.is_a? String and key.size == 16 raise ArgumentError.new("invalid iv") unless iv.is_a? String and iv.size == 16 cipher = OpenSSL::Cipher::Cipher.new( 'AES-128-CBC' ) cipher.decrypt( key, iv ) # NOTE: If the key and iv aren't set this doesn't work correctly. cipher.key = key cipher.iv = iv plaintext = cipher.update( plaintext ) plaintext + cipher.final end #----------------------------------------------------------------------------# ## # Generate and return a message digest for the data from the IO stream # ((|io|)), using the algorithm alg # def Crypto.digest(io, alg = OpenSSL::Digest::SHA1.new) raise ArgumentError.new('io') unless io.kind_of?(IO) or io.kind_of?(StringIO) while not io.eof? alg.update(io.read(BUFFER_SIZE)) end alg.digest end #----------------------------------------------------------------------------# # Return the HMAC SHA1 of _data_ using _key_. def Crypto.hmac_sha1( key, data ) raise ParameterError.new( "key must be a String" ) unless key.is_a? String raise ParameterError.new( "data must be a String" ) unless data.is_a? String md = OpenSSL::Digest::SHA1.new hmac = OpenSSL::HMAC.new( key, md) hmac.update( data ) return hmac.digest end #----------------------------------------------------------------------------# ## # Decrypt the specified cipher text file to create the specified plain text # file. # # The symmetric cipher is AES in CBC mode. 128 bit keys are used. If the plain # text file already exists it will be overwritten. # # ((|key|)) The 128 bit (16 byte) symmetric key. # ((|src|)) The name of the cipher text file to encrypt. # ((|dst|)) The name of the plain text file to create. # ((|iv|)) The 128 bit (16 byte) initialization vector. # def Crypto.encryptfile(src, dst, key, iv) raise ArgumentError.new("invalid file name: #{src}") unless FileTest.exists?(src) raise ArgumentError.new("invalid key") unless key and key.size == 16 raise ArgumentError.new("invalid iv") unless iv and iv.size == 16 cmd = "openssl enc -e -aes-128-cbc -in #{src} -out #{dst} -K #{Format::bin2hex(key)} -iv #{Format::bin2hex(iv)}" result = Kernel::system(cmd) raise "error encrypting file #{src}" unless result end #----------------------------------------------------------------------------# ## # Encrypt _plaintext_ with _key_ and _iv_ using AES-128-CBC. # def Crypto.encryptsym(plaintext, key, iv) raise ArgumentError.new("plaintext must be a String") unless plaintext.kind_of? String raise ArgumentError.new("invalid key") unless ( key.is_a? String and key.size == 16 ) raise ArgumentError.new("invalid iv") unless ( iv.is_a? String and iv.size == 16 ) cipher = OpenSSL::Cipher::Cipher.new( 'AES-128-CBC' ) cipher.encrypt( key, iv ) # NOTE: If the key and iv aren't set this doesn't work correctly. cipher.key = key cipher.iv = iv ciphertext = cipher.update( plaintext ) ciphertext + cipher.final end #----------------------------------------------------------------------------# ## # Generate an initialization vector suitable use with symmetric cipher. # def Crypto.geniv OpenSSL::Cipher::Cipher.new(SYM_ALG).random_iv end #----------------------------------------------------------------------------# ## # Generate a key suitable for use with a symmetric cipher. # def Crypto.gensymkey OpenSSL::Cipher::Cipher.new(SYM_ALG).random_key end #----------------------------------------------------------------------------# ## # Return the public key from the X509 certificate file ((|filename|)). # def Crypto.certfile2pubkey(filename) begin File.open(filename) do |f| return cert2pubkey(f) end rescue Exception => e raise "error reading certificate file #{filename}: #{e.message}" end end #----------------------------------------------------------------------------# def Crypto.cert2pubkey(data) begin return OpenSSL::X509::Certificate.new(data).public_key rescue Exception => e raise "error reading certificate: #{e.message}" end end #----------------------------------------------------------------------------# ## # Sign the data from IO stream or string ((|data|)) using the key in # ((|keyfilename|)). # # Return the signature. # def Crypto.sign(data, keyfilename) raise ArgumentError.new('data') unless data raise ArgumentError.new("invalid file name: #{keyfilename}") unless FileTest.exists?(keyfilename) # Create an IO stream from the data if necessary. io = (data.instance_of?(StringIO) ? data : StringIO.new(data)) sha = OpenSSL::Digest::SHA1.new pk = loadprivkey( keyfilename ) return pk.sign(sha, io.read ) end #------------------------------------------------------------------------------# ## # Generate the SHA1 fingerprint for a PEM-encoded certificate (NOT private key) # Returns the fingerprint in aa:bb:... form # Raises ArgumentError if the fingerprint cannot be obtained # def Crypto.cert_sha1_fingerprint(cert_filename) raise ArgumentError.new('cert_filename is nil') if cert_filename.nil? raise ArgumentError.new("invalid cert file name: #{cert_filename}") unless FileTest.exists?(cert_filename) fingerprint = nil IO.popen("openssl x509 -in #{cert_filename} -noout -sha1 -fingerprint") do |io| out = io.read md = SHA1_FINGERPRINT_REGEX.match(out) if md fingerprint = md[1] end end raise ArgumentError.new("could not generate fingerprint for #{cert_filename}") if fingerprint.nil? return fingerprint end #------------------------------------------------------------------------------# def Crypto.loadprivkey filename begin OpenSSL::PKey::RSA.new( File.open( filename,'r' ) ) rescue Exception => e raise "error reading private key from file #{filename}: #{e.message}" end end #----------------------------------------------------------------------------# ## # XOR the byte string ((|a|)) with the byte string ((|b|)). The operans must # be of the same length. # def Crypto.xor(a, b) raise ArgumentError.new('data lengths differ') unless a.size == b.size xored = String.new a.size.times do |i| xored << (a[i] ^ b[i]) end xored end end ec2-ami-tools-1.4.0.9/lib/ec2/amitools/migratemanifest.rb0000644000000000000000000001775012050102225021505 0ustar rootroot# Copyright 2008-2009 Amazon.com, Inc. or its affiliates. All Rights # Reserved. Licensed under the Amazon Software License (the # "License"). You may not use this file except in compliance with the # License. A copy of the License is located at # http://aws.amazon.com/asl or in the "license" file accompanying this # file. This file is distributed on an "AS IS" BASIS, WITHOUT # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See # the License for the specific language governing permissions and # limitations under the License. require 'ec2/amitools/migratemanifestparameters' require 'ec2/amitools/manifest_wrapper' require 'fileutils' require 'csv' require 'net/http' require 'ec2/amitools/tool_base' require 'ec2/amitools/mapids' MIGRATE_MANIFEST_NAME = "ec2-migrate-manifest" MIGRATE_MANIFEST_MANUAL =< e raise BadManifestError.new(manifest_path, e.message) end unless manifest.authenticate(File.open(user_cert_path)) raise BadManifestError.new(manifest_path, "Manifest fails authentication.") end manifest end #----------------------------------------------------------------------------# def get_mappings(*args) KernelMappings.new(*args) end #----------------------------------------------------------------------------# def map_identifiers(manifest, user, pass, region, kernel_id=nil, ramdisk_id=nil) if region.nil? raise EC2FatalError.new(1, "No region provided, cannot map automatically.") end if manifest.kernel_id.nil? and manifest.ramdisk_id.nil? # Exit early if we have nothing to do return [kernel_id, ramdisk_id] end begin mappings = get_mappings(user, pass, [manifest.kernel_id, manifest.ramdisk_id].compact, region) rescue KernelMappings::MappingError => e raise EC2FatalError.new(7, e.message) end begin if manifest.kernel_id kernel_id ||= mappings[manifest.kernel_id] end if manifest.ramdisk_id ramdisk_id ||= mappings[manifest.ramdisk_id] end warn_about_mappings(mappings.find_missing_targets([kernel_id, ramdisk_id].compact)) rescue KernelMappings::MappingError => e raise EC2FatalError.new(6, e.message) end [kernel_id, ramdisk_id] end #----------------------------------------------------------------------------# def backup_manifest(manifest_path, quiet=false) backup_manifest = "#{manifest_path}.bak" if File::exists?(backup_manifest) raise EC2FatalError.new(2, "Backup file '#{backup_manifest}' already exists. Please delete or rename it and try again.") end $stdout.puts("Backing up manifest...") unless quiet $stdout.puts("Backup manifest at #{backup_manifest}") if @debug FileUtils::copy(manifest_path, backup_manifest) end #----------------------------------------------------------------------------# def build_migrated_manifest(manifest, user_pk_path, kernel_id=nil, ramdisk_id=nil) new_manifest = ManifestV20071010.new() manifest_params = { :name => manifest.name, :user => manifest.user, :image_type => manifest.image_type, :arch => manifest.arch, :reserved => nil, :parts => manifest.parts.map { |part| [part.filename, part.digest] }, :size => manifest.size, :bundled_size => manifest.bundled_size, :user_encrypted_key => manifest.user_encrypted_key, :ec2_encrypted_key => manifest.ec2_encrypted_key, :cipher_algorithm => manifest.cipher_algorithm, :user_encrypted_iv => manifest.user_encrypted_iv, :ec2_encrypted_iv => manifest.ec2_encrypted_iv, :digest => manifest.digest, :digest_algorithm => manifest.digest_algorithm, :privkey_filename => user_pk_path, :kernel_id => kernel_id, :ramdisk_id => ramdisk_id, :product_codes => manifest.product_codes, :ancestor_ami_ids => manifest.ancestor_ami_ids, :block_device_mapping => manifest.block_device_mapping, :bundler_name => manifest.bundler_name, :bundler_version => manifest.bundler_version, :bundler_release => manifest.bundler_release, :kernel_name => manifest.kernel_name, } new_manifest.init(manifest_params) new_manifest end #----------------------------------------------------------------------------# def check_and_warn(manifest, kernel_id, ramdisk_id) if (manifest.kernel_id and kernel_id.nil?) or (manifest.ramdisk_id and ramdisk_id.nil?) message = ["This operation will remove the kernel and/or ramdisk associated with", "the AMI. This may cause the AMI to fail to launch unless you specify", "an appropriate kernel and ramdisk at launch time.", ].join("\n") unless warn_confirm(message) raise EC2StopExecution.new() end end end #----------------------------------------------------------------------------# def warn_about_mappings(problems) return if problems.nil? message = ["The following identifiers do not exist in the target region:", " " + problems.inspect.gsub(/['"]/, '') ].join("\n") unless warn_confirm(message) raise EC2StopExecution.new() end end #----------------------------------------------------------------------------# def migrate_manifest(manifest_path, user_pk_path, user_cert_path, user=nil, pass=nil, use_mapping=true, kernel_id=nil, ramdisk_id=nil, region=nil, quiet=false) manifest = get_manifest(manifest_path, user_cert_path) backup_manifest(manifest_path, quiet) if use_mapping kernel_id, ramdisk_id = map_identifiers(manifest, user, pass, region, kernel_id, ramdisk_id) end check_and_warn(manifest, kernel_id, ramdisk_id) new_manifest = build_migrated_manifest(manifest, user_pk_path, kernel_id, ramdisk_id) File.open(manifest_path, 'w') { |f| f.write(new_manifest.to_s) } $stdout.puts("Successfully migrated #{manifest_path}") unless quiet $stdout.puts("It is now suitable for use in #{region}.") unless quiet end #------------------------------------------------------------------------------# # Overrides #------------------------------------------------------------------------------# def get_manual() MIGRATE_MANIFEST_MANUAL end def get_name() MIGRATE_MANIFEST_NAME end def main(p) migrate_manifest(p.manifest_path, p.user_pk_path, p.user_cert_path, p.user, p.pass, p.use_mapping, p.kernel_id, p.ramdisk_id, p.region) end end #------------------------------------------------------------------------------# # Script entry point. Execute only if this file is being executed. if __FILE__ == $0 ManifestMigrator.new().run(MigrateManifestParameters) end ec2-ami-tools-1.4.0.9/notice.txt0000644000000000000000000000102712050102225014743 0ustar rootrootCopyright 2008 Amazon.com, Inc. or its affiliates. All Rights Reserved. Licensed under the Amazon Software License (the "License"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/asl or in the "license" file accompanying this file. This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.