![\"Screen](\"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2013\/08\/Screen-Shot-2013-08-21-at-12.08.04-PM.png?w=640\")
<\/p>\n<\/p>\n
So the \/etc\/fstab file on your root volume looked like this<\/p>\n
by mistake you deleted the ebs volume that you had mounted on \/mnt\/backups (or whatever folder) and you restarted your ubuntu instance not knowing that if the \/etc\/fstab would break it would not continue to start all the application layer networking services like ssh on port 22…<\/p>\n you can ping the machine, but you can’t ssh, amazon support won’t respond or will tell you to fuck yourself.<\/p>\n you learn that ubuntu has had this bug for a while, but it’s been addressed by passing your volume configuration a nobootwait<\/strong> option.<\/p>\n you wish your \/etc\/fstab looked like this, but you can’t get in, amazon doesn’t give you any other options from their console to go in and solve the problem through a console…<\/p>\n No worries, I have a fix that will let you edit that file, and boot back and try to recover things, you may have lost that ebs volume, but you won’t have to setup this computer again.<\/p>\n 1. Make a snapshot of the root volume on that instance. This will take a while. Disk \/dev\/xvdf doesn't contain a valid partition table<\/strong><\/code><\/p>\n Don’t listen to that last message, you do have a valid partition.<\/p>\n 7. Create a folder where to mount the disk. sudo mkdir \/mnt\/old-volume<\/strong> So the \/etc\/fstab file on your root volume looked like this LABEL=cloudimg-rootfs \/ ext4 defaults 0 0 \/dev\/xvdf \/mnt\/backups auto defaults,comment=cloudconfig 0 2 by mistake you deleted the ebs volume that you had mounted on \/mnt\/backups (or whatever folder) and you restarted your ubuntu instance not knowing that if the \/etc\/fstab would break it would […]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[15,30],"tags":[124,1301,1156,276,1158,1157,1313,1314,1449,1315,1027],"class_list":["post-3127","post","type-post","status-publish","format-standard","hentry","category-code","category-geeklife","tag-administration","tag-amazon-web-services","tag-aws","tag-cloud","tag-ebs","tag-ec2","tag-fstab","tag-it","tag-linux","tag-management","tag-ubuntu"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5Unzf-Or","jetpack-related-posts":[{"id":2734,"url":"https:\/\/www.gubatron.com\/blog\/sample-etcfstab-entry-for-ebs-volume\/","url_meta":{"origin":3127,"position":0},"title":"Sample \/etc\/fstab entry for EBS volume","author":"gubatron","date":"May 23, 2012","format":false,"excerpt":"gid= and uid= are not valid for XFS, pass \"grpid=\" to your fstab entry. \/dev\/xvdf \/media\/ebs\/data xfs defaults,auto,noatime,noexec,grpid=1000 0 0","rel":"","context":"In \"aws\"","block_context":{"text":"aws","link":"https:\/\/www.gubatron.com\/blog\/tag\/aws\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3213,"url":"https:\/\/www.gubatron.com\/blog\/aws-troubleshooting-how-to-fix-a-broken-ebs-volume-bad-superblock-on-xfs\/","url_meta":{"origin":3127,"position":1},"title":"AWS troubleshooting: how to fix a broken EBS volume (bad superblock on xfs)","author":"gubatron","date":"January 19, 2014","format":false,"excerpt":"As great as EBS volumes are on Amazon Web Services, they can break and not ever mount again, even though your data could still be there intact, a simple corruption on the filesystem structure can cause a lot of damage. On this post I teach you how to move all\u2026","rel":"","context":"In "Geeklife"","block_context":{"text":"Geeklife","link":"https:\/\/www.gubatron.com\/blog\/category\/geeklife\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/01\/car-longshot2.jpg?fit=720%2C482&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/01\/car-longshot2.jpg?fit=720%2C482&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/01\/car-longshot2.jpg?fit=720%2C482&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/01\/car-longshot2.jpg?fit=720%2C482&ssl=1&resize=700%2C400 2x"},"classes":[]},{"id":3977,"url":"https:\/\/www.gubatron.com\/blog\/how-to-resize-aws-ec2-ebs-root-partition-without-rebooting-in-3-steps\/","url_meta":{"origin":3127,"position":2},"title":"How to resize AWS ec2 ebs root partition without rebooting in 3 steps","author":"gubatron","date":"September 14, 2022","format":false,"excerpt":"Go to the AWS EBS dashboard and modify the volume size. Might be good to create a snapshot of it for safety but haven't really failed ever doing this. # 1. Check the device of your partition $ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS loop0 7:0 0\u2026","rel":"","context":"In "Code"","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2022\/09\/xai-tmp-imgen-db8aab31-34e5-4be7-b882-a7e87a2836f9.jpg?fit=1091%2C655&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":1092,"url":"https:\/\/www.gubatron.com\/blog\/preparing-your-primary-mount-and-swap-with-fdisk\/","url_meta":{"origin":3127,"position":3},"title":"Preparing your primary mount and swap with fdisk","author":"gubatron","date":"January 10, 2009","format":false,"excerpt":"I have 4 big drives on a new machine, each can hold up to 2Tb of data, at first I thought I'd use the first drive for the OS and the other 3 for a RAID5 (software controlled) Then after I had installed the operating system, I decided it was\u2026","rel":"","context":"In "Geeklife"","block_context":{"text":"Geeklife","link":"https:\/\/www.gubatron.com\/blog\/category\/geeklife\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3370,"url":"https:\/\/www.gubatron.com\/blog\/how-to-resize-an-ebs-xfs-formatted-partition\/","url_meta":{"origin":3127,"position":4},"title":"How to resize an EBS (xfs formatted) partition","author":"gubatron","date":"December 20, 2014","format":false,"excerpt":"First of all, create a snapshot of your EBS volume. Then out of that snapshot you will be able to create your new volume. However, when you detach the old one from your instance and attach the new one, you will still see the old available space with df look\u2026","rel":"","context":"In "Code"","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=350%2C200","width":350,"height":200,"srcset":"https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=350%2C200 1x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=525%2C300 1.5x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=700%2C400 2x, https:\/\/i0.wp.com\/www.gubatron.com\/blog\/wp-content\/uploads\/2014\/12\/Screen-Shot-2016-01-02-at-4.42.45-PM.png?fit=1200%2C420&ssl=1&resize=1050%2C600 3x"},"classes":[]},{"id":2745,"url":"https:\/\/www.gubatron.com\/blog\/ubuntu-mongodb-wont-start-fixed\/","url_meta":{"origin":3127,"position":5},"title":"Ubuntu: mongodb won’t start [FIXED]","author":"gubatron","date":"May 23, 2012","format":false,"excerpt":"So you decided to change the default location of your mongodb data path on mongodb.conf but mongo won't start and you get an error similar to this [bash] Wed May 23 04:17:35 [initandlisten] MongoDB starting : pid=1214 port=27017 dbpath=\/media\/ebs\/data 64-bit host=domU-12-31-38-00-78-DB Wed May 23 04:17:35 [initandlisten] db version v2.0.4, pdfile\u2026","rel":"","context":"In \"mongo\"","block_context":{"text":"mongo","link":"https:\/\/www.gubatron.com\/blog\/tag\/mongo\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/3127","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/comments?post=3127"}],"version-history":[{"count":10,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/3127\/revisions"}],"predecessor-version":[{"id":3138,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/3127\/revisions\/3138"}],"wp:attachment":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/media?parent=3127"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/categories?post=3127"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/tags?post=3127"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}LABEL=cloudimg-rootfs \/ ext4 defaults 0 0
\n\/dev\/xvdf \/mnt\/backups auto defaults,comment=cloudconfig 0 2
\n<\/code><\/p>\nLABEL=cloudimg-rootfs \/ ext4 defaults 0 0
\n\/dev\/xvdf \/mnt\/backups auto defaults,nobootwait<\/strong>,comment=cloudconfig 0 2
\n<\/code><\/p>\n
\n2. Make a new ebs volume of that snapshot and put it on the zone where the ec2 instance lives.
\n3. Create an identical temporary new ec2 instance on the same zone.
\n4. Attach the snapshot volume you created on step 2 to the new instance.
\n5. ssh to the new machine.
\n6. sudo fdisk -l<\/strong>, you should see all the attached devices, you will see something like this referring to the attached ebs<\/p>\nDisk \/dev\/xvdf: 8589 MB, 8589934592 bytes<\/strong>
\n255 heads, 63 sectors\/track, 1044 cylinders, total 16777216 sectors
\nUnits = sectors of 1 * 512 = 512 bytes
\nSector size (logical\/physical): 512 bytes \/ 512 bytes
\nI\/O size (minimum\/optimal): 512 bytes \/ 512 bytes
\nDisk identifier: 0x00000000<\/p>\n
\n8. Mount it sudo mount -t auto \/dev\/xvdf \/mnt\/old-volume<\/strong>
\n9. Get into \/mnt\/old-volume\/etc\/fstab<\/strong> and fix it.
\n10. Unmount \/mnt\/old-volume<\/strong>, turn off the instance, detach the repaired volume.
\n11. Turn off the original instance, detach the broken root volume (at \/dev\/sda1)
\n12. Attach the repaired volume to the original instance under \/dev\/sda1<\/strong>
\n13. Start the original instance.
\n14. ssh to it. (it will have a new ip address, make sure to update your DNS or load balancing entries)
\n15. Terminate the temporary instance and all the volumes that you won’t need.
\n16. Get to work.
\n17. Leave a tip below. \ud83d\ude09<\/p>\n","protected":false},"excerpt":{"rendered":"