Monday, August 31, 2015

mysql - Lower Maximum Possible Memory Usage



I've been using MySQLTuner to optimize my database, and the only problem I am having is:





[!!] Maximum possible memory usage:
14.8G (126% of installed RAM)




I'm not sure what I should be changing to get this down under my 12G of RAM. I have a database with 110 tables, 10,000,000 rows (growing fairly quickly), and an average of 250 users online. My my.cnf has looks like:




query_cache_limit=1M




query_cache_size=128M



query_cache_type=1



max_user_connections=2000



max_connections=2000



interactive_timeout=100




wait_timeout=100



connect_timeout=100



thread_cache_size=128



key_buffer=16M



join_buffer=1M




max_allowed_packet=16M



table_cache=15360



record_buffer=1M



sort_buffer_size=4M



read_buffer_size=2M




max_connect_errors=10



thread_concurrency=8



myisam_sort_buffer_size=64M



server-id=1



Answer




The memory usage of MySQL is quite hard to predict. I would recommend having a look at http://www.mysqlperformanceblog.com/2006/05/17/mysql-server-memory-usage/ which gives an introduction to what is using memory in your installation. But to summarize:



"In fact typical server with 8GB of memory will often run with maximum theoretical memory usage of 100GB or more."



Or in other words, don't worry too much about the maximum possible memory usage unless you start seeing your server swap.


security - Silently Skip Windows 7 UAC Prompts?


NOTE: I am not trying to disable UAC.


I have a Windows 7 machine where I've added one administrator account and one regular user account. When using the regular user account, a UAC dialog will show up having a password prompt when an application requires administrative privileges. Entering the administrator's password will allow the action to continue, but this is not the behavior that I require. Instead of prompting for the password, I would like the action to either silently fail (with the action logged in the system log) or show a permission denied dialog.


Is this possible? If so, how do I do it?


Answer



You can change how UAC acts in this scenario by using Security Policies:


Instructions from here:



Follow these steps to eliminate UAC prompts for standard users:



  • Click on the start menu, type "secpol.msc" and hit Enter.

  • When the "Local Security Policy" editor ("snap-in") opens, double-click on "Local Policies" -for those affecting this particular
    machine- and double-click on "Security Options".

  • Scroll down until you see a node labeled "User Account Control: Behavior of the Elevation Prompt for Standard Users" (careful, another
    policy starts with the same name, but actually affects computer
    administrators!) Double-click on it and the following dialog will
    open:


enter image description here



  • The default behavior for standard users, selected in the dropdown menu pictured above, is "Prompt for credentials": whenever elevated
    privileges are required, they'll need to select an administrator and
    enter its password in a UAC prompt to proceed with the action.

  • Click on the dropdown, and select the "Automatically deny elevation requests" option instead; click "OK" to apply the new setting: from
    this point on, standard users won't event get a UAC prompt when they
    attempt any task that needs administrative permissions.



See the link for extended information.


windows - How to undo 'create partition primary' command of diskpart

I was trying to recover my pendrive as it was showing 0 bytes allocated space, and I came across some information about using the diskpart command to format it.



For the worse I selected a wrong volume and started the process; I selected my hard drive and typed in 'create partition primary'. Now I cannot see my drives except for the local disk: C, in which my OS is installed. :( Please help me; I have no prior knowledge about this, yet I've toiled my hands and now I am stuck ... the data is very important ...



Thanks in advance for your help!




Email removed

hardware failure - Unable to create a Windows 7 system image of a failing hard drive

The hard disk of my one year old T400 Thinkpad has started failing periodic hardware tests. I get a "Targeted Read Test Failed" error. The "SMART short self test" times out.



I am now trying to create a Windows 7 System image of the hard disk but it fails without giving any specific error messages. I tried using Comodo Backup but got an error (code 101117) there as well.



I have copied the important files in Dropbox but would like to take a full System backup as I have plenty of software installed on the machine.



Does anyone know why this is happening and how I can take a backup of the system image ?

Why does this work: Windows same local admin username and password, able to access other computer?




I've two machines MachineA and MachineB. Both have two local accounts which have same username . Both accounts are local admin on those machines.



If they have same password, I'm able to access \\MachineA\C$ from MachineB.



But if I change any one password above doesn't work. And it prompts for username and password. Now if I provide MachineA\username and password, it then connects.



Why isn't there a prompt for password when passwords are same? I expected passwords to be hashed with random salt, how does windows know passwords are same and authorize access?



These machines are running Windows Server 2008 R2.




PS. I've created following reg key to enable drive access
HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\LocalAccountTokenFilterPolicy to 1


Answer



After you log in, Windows takes your entered password and stores its LM and NT hashes in kernel memory, in order to provide this single-signon function on workgroups and NT4 domains. (On Active Directory domains, the Kerberos TGT obtained during login is stored as well.)



User-space programs cannot access these hashes directly1, but they can ask the kernel to connect to a SMB server using the stored password/ticket, or to perform NTLM challenge-response or Kerberos authentication on behalf of the program (for other protocols such as IMAP or HTTP).



Also, Windows NT uses unsalted MD4 hashes for both local authentication and NTLM, but this is not important – it could just as well keep the plain-text password in memory if it was necessary.







1 There is a tool WCE which, given sufficient privileges, can examine kernel memory and display the stored credentials.


bios - What exactly is "UEFI with CSM" boot mode?

Under this question, I was told that "UEFI with CSM" is essentially BIOS mode. It made me quite confused because what I've learnt over the years is far from that.



As per my observation, I think CSM is a special mechanism that allows previous "not canonical" UEFI OS to boot.



Here's my observation:




  1. The full name for UEFI with CSM is





    UEFI with Compatibility Support Module




    which as its name suggests, is a "supprt module" for UEFI.


  2. I have three partitions on the primary drive in my computer. The drive is formatted to GPT and the three partitions are one ESP (EFI System Partition) and two Primary (Windows 10/Windows 7 each one). There are three options for "Boot Mode" in the motherboard settings, "Legacy", "UEFI" and "UEFI with CSM".



    If it's set to "Legacy", the computer tells me "No operating system found". That's correct. If it's set to "UEFI with CSM", both Windows boots normally. If it's set to "UEFI", then only Windows 10 can boot. From the safe boot log I can see Windows 7 gets stuck at classpnp.sys.



    I tried removing \Windows\System32\winload.exe from both volumes (I did that in a Windows To Go environment), and nothing went wrong, so I restored the file.


  3. I plugged in another drive in MBR, containint two partitions. One primary active with FAT32 and another primary with NTFS, with another Windows 7 installed. It seems I however just can't make it boot if the MB settings is set to either "UEFI" or "UEFI with CSM", but it boots perfectly when set to "Legacy". If I edit the BCD in the EFI partition on the primary disk to add an entry for this experimental Windows 7 installation, it boots under "UEFI with CSM", but still gets stuck at classpnp.sys in "UEFI".



  4. I have another computer with a MSI B85 motherboard that has a switch named "CSM". If it's enabled then there are two options available in "Boot Mode", "UEFI" and "Legacy". If it's disabled then Boot Mode is locked to UEFI. In that case "UEFI" mode with CSM enabled allows Windows 7 to boot, but it won't boot with CSM disabled.




The above experiments made me believe "UEFI with CSM" is a special mode of EFI, so why am I still told it's BIOS?

Dual Boot System - Win7/Win8 - Remove Win7, convert Win8 Logical Part. to Primary

I can not find an answer to this - even from several people I know on the tech side. Dual booting Windows 7 and 8. Windows 7 is the OEM Gateway box. My drive looks like this:


PQService (Primary) System Reserved (Primary, active) Windows 7 (Primary) Windows 8 (Boot, Crash dump, LOGICAL)


I want to remove Win7 and convert the Win8 Logical to Primary. I was thinking I could remove the PQService, which is a backup for Win7 but I'm not sure. The small, 100MB System reserved is the only partition which shows active, making this even more confusing.


I'd like to end up with: System Reserved, Windows 8


Any thoughts on how to safely get there? Thank you!

networking - How does IPv4 Subnetting Work?





This is a Canonical Question about IPv4 Subnets.



Related:






How does Subnetting Work, and How do you do it by hand or in your head? Can someone explain both conceptually and with several examples? subnetting homework questions, so we could use an answer to point them itself.





  • If I have a network, how do I figure
    out how to split it up?

  • If I am given a netmask, how do I
    know what the network Range is for
    it?

  • Sometimes there is a slash followed
    by a number, what is that number?

  • Sometimes there is a subnet mask, but also a wildcard mask, they seem like the same thing but they are different?


  • Someone mentioned something about knowing binary for this?


Answer



IP subnets exist to allow routers to choose appropriate destinations for packets. You can use IP subnets to break up larger networks for logical reasons (firewalling, etc), or physical need (smaller broadcast domains, etc).



Simply put, though, IP routers use your IP subnets to make routing decisions. Understand how those decisions work, and you can understand how to plan IP subnets.



Counting to 1



If you are already fluent in binary (base 2) notation you can skip this section.




For those of you who are left: Shame on you for not being fluent in binary notation!



Yeah-- that may be a bit harsh. It's really, really easy to learn to count in binary, and to learn shortcuts to convert binary to decimal and back. You really should know how to do it.



Counting in binary is so simple because you only have to know how to count to 1!



Think of a car's "odometer", except that unlike a traditional odometer each digit can only count up to 1 from 0. When the car is fresh from the factory the odometer reads "00000000".



When you've driven your first mile the odometer reads "00000001". So far, so good.




When you've driven your second mile the first digit of the odometer rolls back over to "0" (since it's maximum value is "1") and the second digit of the odometer rolls over to "1", making the odometer read "00000010". This looks like the number 10 in decimal notation, but it's actually 2 (the number of miles you've driven the car so far) in binary notation.



When you've driven the third mile the odometer reads "00000011", since the first digit of the odometer turns again. The number "11", in binary notation, is the same as the decimal number 3.



Finally, when you've driven your fourth mile both digits (which were reading "1" at the end of the third mile) roll back over to zero position, and the 3rd digit rolls up to the "1" position, giving us "00000100". That's the binary representation of the decimal number 4.



You can memorize all of that if you want, but you really only need to understand how the little odometer "rolls over" as the number it's counting gets bigger. It's exactly the same as a traditional decimal odometer's operation, except that each digit can only be "0" or "1" on our fictional "binary odometer".



To convert a decimal number to binary you could roll the odometer forward, tick by tick, counting aloud until you've rolled it a number of times equal to the decimal number you want to convert to binary. Whatever is displayed on the odometer after all that couting and rolling would be the binary representation of the decimal number you counted up to.




Since you understand how the odometer rolls forward you'll also understand how it rolls backward, too. To convert a binary number displayed on the odometer back to decimal you could roll the odometer back one tick at a time, counting aloud until the odometer reads "00000000". When all that counting and rolling is done, the last number you say aloud would be the decimal representation of the binary number the odometer started with.



Converting values between binary and decimal this way would be very tedious. You could do it, but it wouldn't be very efficient. It's easier to learn a little algorithm to do it faster.



A quick aside: Each digit in a binary number is known as a "bit". That's "b" from "binary" and "it" from "digit". A bit is a binary digit.



Converting a binary number like, say, "1101011" to decimal is a simple process with a handy little algorithm.



Start by counting the number of bits in the binary number. In this case, there are 7. Make 7 divisions on a sheet of paper (in your mind, in a text file, etc) and begin filling them in from right to left. In the rightmost slot, enter the number "1", because we'll always start with "1". In the next slot to the left enter double the value in the slot to the right (so, "2" in the next one, "4" in the next one) and continue until all the slots are full. (You'll end up memorizing these numbers, which are the powers of 2, as you do this more and more. I'm alright up to 131,072 in my head but I usually need a calculator or paper after that).




So, you should have the following on your paper in your little slots.



 64    |    32    |    16    |    8    |    4    |    2    |    1    |


Transcribe the bits from the binary number below the slots, like so:



 64    |    32    |    16    |    8    |    4    |    2    |    1    |
1 1 0 1 0 1 1



Now, add some symbols and compute the answer to the problem:



 64    |    32    |    16    |    8    |    4    |    2    |    1    |
x 1 x 1 x 0 x 1 x 0 x 1 x 1
--- --- --- --- --- --- ---
+ + + + + + =



Doing all the math, you should come up with:



 64    |    32    |    16    |    8    |    4    |    2    |    1    |
x 1 x 1 x 0 x 1 x 0 x 1 x 1
--- --- --- --- --- --- ---
64 + 32 + 0 + 8 + 0 + 2 + 1 = 107


That's got it. "1101011" in decimal is 107. It's just simple steps and easy math.




Converting decimal to binary is just as easy and is the same basic algorithm, run in reverse.



Say that we want to convert the number 218 to binary. Starting on the right of a sheet of paper, write the number "1". To the left, double that value (so, "2") and continue moving toward the left of the paper doubling the last value. If the number you are about to write is greater than the number being converted stop writing. otherwise, continue doubling the prior number and writing. (Converting a big number, like 34,157,216,092, to binary using this algorithm can be a bit tedious but it's certainly possible.)



So, you should have on your paper:



 128    |    64    |    32    |    16    |    8    |    4    |    2    |    1    |


You stopped writing numbers at 128 because doubling 128, which would give you 256, would be large than the number being converted (218).




Beginning from the leftmost number, write "218" above it (128) and ask yourself: "Is 218 larger than or equal to 128?" If the answer is yes, scratch a "1" below "128". Above "64", write the result of 218 minus 128 (90).



Looking at "64", ask yourself: "Is 90 larger than or equal to 64?" It is, so you'd write a "1" below "64", then subtract 64 from 90 and write that above "32" (26).



When you get to "32", though, you find that 32 is not greater than or equal to 26. In this case, write a "0" below "32", copy the number (26) from above 32" to above "16" and then continue asking yourself the same question with the rest of the numbers.



When you're all done, you should have:



 218         90         26         26        10         2         2         0

128 | 64 | 32 | 16 | 8 | 4 | 2 | 1 |
1 1 0 1 1 0 1 0


The numbers at the top are just notes used in computation and don't mean much to us. At the bottom, though, you see a binary number "11011010". Sure enough, 218, converted to binary, is "11011010".



Following these very simple procedures you can convert binary to decimal and back again w/o a calculator. The math is all very simple and the rules can be memorized with just a bit of practice.



Splitting Up Addresses




Think of IP routing like pizza delivery.



When you're asked to deliver a pizza to "123 Main Street" it's very clear to you, as a human, that you want to go to the building numbered "123" on the street named "Main Street". It's easy to know that you need to go to the 100-block of Main Street because the building number is between 100 and 199 and most city blocks are numbered in hundreds. You "just know" how to split the address up.



Routers deliver packets, not pizza. Their job is the same as a pizza driver: To get the cargo (packets) as close to the destination as possible. A router is connected to two or more IP subnets (to be at all useful). A router must examine destination IP addresses of packets and break those destination addresses up into their "street name" and "building number" components, just like the pizza driver, to make decisions about delivery.



Each computer (or "host") on an IP network is configured with a unique IP address and subnet mask. That IP address can be divided up into a "building number" component (like "123" in the example above) called the "host ID" and a "street name" component (like "Main Street" in the example above) called the "network ID". For our human eyes, it's easy to see where the building number and the street name are in "123 Main Street", but harder to see that division in "10.13.216.41 with a subnet mask of 255.255.192.0".



IP routers "just know" how to split up IP addresses into these component parts to make routing decisions. Since understanding how IP packets are routed hinges on understanding this process we need to know how to break up IP addresses, too. Fortunately, extracting the host ID and the network ID out of an IP address and subnet mask is actually pretty easy.




Start by writing out the IP address in binary (use a calculator if you haven't learned to do this in your head just yet, but make a note learn how to do it-- it's really, really easy and impresses the opposite sex at parties):



      10.      13.     216.      41
00001010.00001101.11011000.00101001


Write out the subnet mask in binary, too:



     255.     255.     192.       0
11111111.11111111.11000000.00000000



Written side-by-side, you can see that the point in the subnet mask where the "1's" stop "lines up" to a point in the IP address. That's the point that the network ID and the host ID split. So, in this case:



      10.      13.     216.      41
00001010.00001101.11011000.00101001 - IP address
11111111.11111111.11000000.00000000 - subnet mask
00001010.00001101.11000000.00000000 - Portion of IP address covered by 1's in subnet mask, remaining bits set to 0
00000000.00000000.00011000.00101001 - Portion of IP address covered by 0's in subnet mask, remaining bits set to 0



Routers use the subnet mask to "mask out" the bits covered by 1's in the IP address (replacing the bits that are not "masked out" with 0's) to extract the network ID:



      10.      13.     192.       0
00001010.00001101.11000000.00000000 - Network ID


Likewise, by using the subnet mask to "mask out" the bits covered by 0's in the IP address (replacing the bits that are not "masked out" with 0's again) a router can extract the host ID:



       0.       0.      24.      41

00000000.00000000.00011000.00101001 - Portion of IP address covered by 0's in subnet mask, remaining bits set to 0


It's not as easy for our human eyes to see the "break" between the network ID and the host ID as it is between the "building number" and the "street name" in physical addresses during pizza delivery, but the ultimate effect is the same.



Now that you can split up IP addresses and subnet masks into host ID's and network ID's you can route IP just like a router does.



More Terminology



You're going to see subnet masks written all over the Internet and throughout the rest of this answer as (IP/number). This notation is known as "Classless Inter-Domain Routing" (CIDR) notation. "255.255.255.0" is made up of 24 bits of 1's at the beginning, and it's faster to write that as "/24" than as "255.255.255.0". To convert a CIDR number (like "/16") to a dotted-decimal subnet mask just write out that number of 1's, split it into groups of 8 bits, and convert it to decimal. (A "/16" is "255.255.0.0", for instance.)




Back in the "old days", subnet masks weren't specified, but rather were derived by looking at certain bits of the IP address. An IP address starting with 0 - 127, for example, had an implied subnet mask of 255.0.0.0 (called a "class A" IP address).



These implied subnet masks aren't used today and I don't recommend learning about them anymore unless you have the misfortune of dealing with very old equipment or old protocols (like RIPv1) that don't support classless IP addressing. I'm not going to mention these "classes" of addresses further because it's inapplicable today and can be confusing.



Some devices use a notation called "wildcard masks". A "wildcard mask" is nothing more than a subnet mask with all 0's where there would be 1's, and 1's where there would be 0's. The "wildcard mask" of a /26 is:



 11111111.11111111.11111111.11000000 - /26 subnet mask
00000000.00000000.00000000.00111111 - /26 "wildcard mask"



Typically you see "wildcard masks" used to match host IDs in access-control lists or firewall rules. We won't discuss them any further here.



How a Router Works



As I've said before, IP routers have a similar job to a pizza delivery driver in that they need to get their cargo (packets) to its destination. When presented with a packet bound for address 192.168.10.2, an IP router needs to determine which of its network interfaces will best get that packet closer to its destination.



Let's say that you are an IP router, and you have interfaces connected to you numbered:





  • Ethernet0 - 192.168.20.1, subnet mask /24

  • Ethernet1 - 192.168.10.1, subnet mask /24



If you receive a packet to deliver with a destination address of "192.168.10.2", it's pretty easy to tell (with your human eyes) that the packet should be sent out the interface Ethernet1, because the Ethernet1 interface address corresponds to the packet's destination address. All the computers attached to the Ethernet1 interface will have IP addresses starting with "192.168.10.", because the network ID of the IP address assigned to your interface Ethernet1 is "192.168.10.0".



For a router, this route selection process is done by building a routing table and consulting the table each time a packet is to be delivered. A routing table contains network ID and destination interface names. You already know how to obtain a network ID from an IP address and subnet mask, so you're on your way to building a routing table. Here's our routing table for this router:




  • Network ID: 192.168.20.0 (11000000.10101000.00010100.00000000) - 24 bit subnet mask - Interface Ethernet0


  • Network ID: 192.168.10.0 (11000000.10101000.00001010.00000000) - 24 bit subnet mask - Interface Ethernet1



For our incoming packet bound for "192.168.10.2", we need only convert that packet's address to binary (as humans-- the router gets it as binary off the wire to begin with) and attempt to match it to each address in our routing table (up to the number of bits in the subnet mask) until we match an entry.




  • Incoming packet destination: 11000000.10101000.00001010.00000010



Comparing that to the entries in our routing table:




11000000.10101000.00001010.00000010 - Destination address for packet
11000000.10101000.00010100.00000000 - Interface Ethernet0
!!!!!!!!.!!!!!!!!.!!!????!.xxxxxxxx - ! indicates matched digits, ? indicates no match, x indicates not checked (beyond subnet mask)

11000000.10101000.00001010.00000010 - Destination address for packet
11000000.10101000.00001010.00000000 - Interface Ethernet1, 24 bit subnet mask
!!!!!!!!.!!!!!!!!.!!!!!!!!.xxxxxxxx - ! indicates matched digits, ? indicates no match, x indicates not checked (beyond subnet mask)



The entry for Ethernet0 matches the first 19 bits fine, but then stops matching. That means it's not the proper destination interface. You can see that the interface Ethernet1 matches 24 bits of the destination address. Ah, ha! The packet is bound for interface Ethernet1.



In a real-life router, the routing table is sorted in such a manner that the longest subnet masks are checked for matches first (i.e. the most specific routes), and numerically so that as soon as a match is found the packet can be routed and no further matching attempts are necessary (meaning that 192.168.10.0 would be listed first and 192.168.20.0 would never have been checked). Here, we're simplifying that a bit. Fancy data structures and algorithms make faster IP routers, but simple algorithms will produce the same results.



Static Routes



Up to this point, we've talked about our hypothetical router as having networks directly connected to it. That's not, obviously, how the world really works. In the pizza-driving analogy, sometimes the driver isn't allowed any further into the building than the front desk, and has to hand-off the pizza to somebody else for delivery to the final recipient (suspend your disbelief and bear with me while I stretch my analogy, please).



Let's start by calling our router from the earlier examples "Router A". You already know RouterA's routing table as:





  • Network ID: 192.168.20.0 (11000000.10101000.00010100.00000000) - subnet mask /24 - Interface RouterA-Ethernet0

  • Network ID: 192.168.10.0 (11000000.10101000.00001010.00000000) - subnet mask /24 - Interface RouterA-Ethernet1



Suppose that there's another router, "Router B", with the IP addresses 192.168.10.254/24 and 192.168.30.1/24 assigned to its Ethernet0 and Ethernet1 interfaces. It has the following routing table:




  • Network ID: 192.168.10.0 (11000000.10101000.00001010.00000000) - subnet mask /24 - Interface RouterB-Ethernet0

  • Network ID: 192.168.30.0 (11000000.10101000.00011110.00000000) - subnet mask /24 - Interface RouterB-Ethernet1




In pretty ASCII art, the network looks like this:



               Interface                      Interface
Ethernet1 Ethernet1
192.168.10.1/24 192.168.30.254/24
__________ V __________ V
| | V | | V
----| ROUTER A |------- /// -------| ROUTER B |----

^ |__________| ^ |__________|
^ ^
Interface Interface
Ethernet0 Ethernet0
192.168.20.1/24 192.168.10.254/24


You can see that Router B knows how to "get to" a network, 192.168.30.0/24, that Router A knows nothing about.



Suppose that a PC with the IP address 192.168.20.13 attached to the network connected to router A's Ethernet0 interface sends a packet to Router A for delivery. Our hypothetical packet is destined for the IP address 192.168.30.46, which is a device attached to the network connected to the Ethernet1 interface of Router B.




With the routing table shown above, neither entry in Router A's routing table matches the destination 192.168.30.46, so Router A will return the packet to the sending PC with the message "Destination network unreachable".



To make Router A "aware" of the existence of the 192.168.30.0/24 network, we add the following entry to the routing table on Router A:




  • Network ID: 192.168.30.0 (11000000.10101000.00011110.00000000) - subnet mask /24 - Accessible via 192.168.10.254



In this way, Router A has a routing table entry that matches the 192.168.30.46 destination of our example packet. This routing table entry effectively says "If you get a packet bound for 192.168.30.0/24, send it on to 192.168.10.254 because he knows how to deal with it." This is the analogous "hand-off the pizza at the front desk" action that I mentioned earlier-- passing the packet on to somebody else who knows how to get it closer to its destination.




Adding an entry to a routing table "by hand" is known as adding a "static route".



If Router B wants to deliver packets to the 192.168.20.0 subnet mask 255.255.255.0 network, it will need an entry in its routing table, too:




  • Network ID: 192.168.20.0 (11000000.10101000.00010100.00000000) - subnet mask /24 - Accessible via: 192.168.10.1 (Router A's IP address in the 192.168.10.0 network)



This would create a path for delivery between the 192.168.30.0/24 network and the 192.168.20.0/24 network across the 192.168.10.0/24 network between these routers.




You always want to be sure that routers on both sides of such an "interstitial network" have a routing table entry for the "far end" network. If router B in our example didn't have a routing table entry for "far end" network 192.168.20.0/24 attached to router A our hypothetical packet from the PC at 192.168.20.13 would get to the destination device at 192.168.30.46, but any reply that 192.168.30.46 tried to send back would be returned by router B as "Destination network unreachable." One-way communication is generally not desirable. Always be sure you think about traffic flowing in both directions when you think about communication in computer networks.



You can get a lot of mileage out of static routes. Dynamic routing protocols like EIGRP, RIP, etc, are really nothing more than a way for routers to exchange routing information between each other that could, in fact, be configured with static routes. One large advantage to using dynamic routing protocols over static routes, though, is that dynamic routing protocols can dynamically change the routing table based on network conditions (bandwidth utilization, an interface "going down", etc) and, as such, using a dynamic routing protocol can result in a configuration that "routes around" failures or bottlenecks in the network infrastructure. (Dynamic routing protocols are WAY outside the scope of this answer, though.)



You Can't Get There From Here



In the case of our example Router A, what happens when a packet bound for "172.16.31.92" comes in?



Looking at the Router A routing table, neither destination interface or static route matches the first 24 bits of 172.18.31.92 (which is 10101100.00010000.00011111.01011100, BTW).




As we already know, Router A would return the packet to the sender via a "Destination network unreachable" message.



Say that there's another router (Router C) sitting at the address "192.168.20.254". Router C has a connection to the Internet!



                              Interface                      Interface                      Interface
Ethernet1 Ethernet1 Ethernet1
192.168.20.254/24 192.168.10.1/24 192.168.30.254/24
__________ V __________ V __________ V
(( heap o )) | | V | | V | | V

(( internet )) ----| ROUTER C |------- /// -------| ROUTER A |------- /// -------| ROUTER B |----
(( w00t! )) ^ |__________| ^ |__________| ^ |__________|
^ ^ ^
Interface Interface Interface
Ethernet0 Ethernet0 Ethernet0
10.35.1.1/30 192.168.20.1/24 192.168.10.254/24


It would be nice if Router A could route packets that do not match any local interface up to Router C such that Router C can send them on to the Internet. Enter the "default gateway" route.




Add an entry at the end of our routing table like this:




  • Network ID: 0.0.0.0 (00000000.00000000.00000000.00000000) - subnet mask /0 - Destination router: 192.168.20.254



When we attempt to match "172.16.31.92" to each entry in the routing table we end up hitting this new entry. It's a bit perplexing, at first. We're looking to match zero bits of the destination address with... wait... what? Matching zero bits? So, we're not looking for a match at all. This routing table entry is saying, basically, "If you get here, rather than giving up on delivery, send the packet on to the router at 192.168.20.254 and let him handle it".



192.168.20.254 is a destination we DO know how to deliver a packet to. When confronted with a packet bound for a destination for which we have no specific routing table entry this "default gateway" entry will always match (since it matches zero bits of the destination address) and gives us a "last resort" place that we can send packets for delivery. You'll sometimes hear the default gateway called the "gateway of last resort."




In order for a default gateway route to be effective it must refer to a router that is reachable using the other entries in the routing table. If you tried to specify a default gateway of 192.168.50.254 in Router A, for example, delivery to such a default gateway would fail. 192.168.50.254 isn't an address that Router A knows how to deliver packets to using any of the other routes in its routing table, so such an address would be ineffective as a default gateway. This can be stated concisely: The default gateway must be set to an address already reachable by using another route in the routing table.



Real routers typically store the default gateway as the last route in their routing table such that it matches packets after they've failed to match all other entries in the table.



Urban Planning and IP Routing



Breaking up a IP subnet into smaller IP subnets is lke urban planning. In urban planning, zoning is used to adapt to natural features of the landscape (rivers, lakes, etc), to influence traffic flows between different parts of the city, and to segregate different types of land-use (industrial, residential, etc). IP subnetting is really much the same.



There are three main reasons why you would subnet a network:





  • You may want to communicate across different unlike communication media. If you have a T1 WAN connection between two buildings IP routers could be placed on the ends of these connections to facilitate communication across the T1. The networks on each end (and possibly the "interstitial" network on the T1 itself) would be assigned to unique IP subnets so that the routers can make decisions about which traffic should be sent across the T1 line.


  • In an Ethernet network, you might use subnetting to limit the amount of broadcast traffic in a given portion of the network. Application-layer protocols use the broadcast capability of Ethernet for very useful purposes. As you get more and more hosts packed into the same Ethernet network, though, the percentage of broadcast traffic on the wire (or air, in wireless Ethernet) can increase to such a point as to create problems for delivery of non-broadcast traffic. (In the olden days, broadcast traffic could overwhelm the CPU of hosts by forcing them to examine each broadcast packet. That's less likely today.) Excessive traffic on switched Ethernet can also come in form of "flooding of frames to unknown destinations". This condition is caused by an Ethernet switch being unable to keep track of every destination on the network and is the reason why switched Ethernet networks can't scale to an infinite number of hosts. The effect of flooding of frames to unknown destinations is similar to the the effect of excess broadcast traffic, for the purposes of subnetting.


  • You may want to "police" the types of traffic flowing between different groups of hosts. Perhaps you have print server devices and you only want authorized print queuing server computers to send jobs to them. By limiting the traffic allowed to flow to the print server device subnet users can't configure their PCs to talk directly to the print server devices to bypass print accounting. You might put the print server devices into a subnet all to themselves and create a rule in the router or firewall attached to that subnet to control the list of hosts permitted to send traffic to the print server devices. (Both routers and firewalls can typically make decisions about how or whether to deliver a packet based on the source and destination addresses of the packet. Firewalls are typically a sub-species of router with an obsessive personality. They can be very, very concerned about the payload of packets, whereas routers typically disregard payloads and just deliver the packets.)




In planning a city, you can plan how streets intersect with each other, and can use turn-only, one-way, and dead-end streets to influence traffic flows. You might want Main Street to be 30 blocks long, with each block having up to 99 buildings each. It's pretty easy to plan your street numbering such that each block in Main Street has a range of street numbers increasing by 100 for each block. It's very easy to know what the "starting number" in each subsequent block should be.



In planning IP subnets, you're concerned with building the right number of subnets (streets) with the right number of available host ID's (building numbers), and using routers to connect the subnets to each other (intersections). Rules about allowed source and destination addresses specified in the routers can further control the flow of traffic. Firewalls can act like obsessive traffic cops.




For the purposes of this answer, building our subnets is our only major concern. Instead of working in decimal, as you would with urban planning, you work in binary to describe the bounds of each subnet.



Continued on: How does IPv4 Subnetting Work?



(Yes ... we reached the maximum size of an answer (30000 characters).)


Windows 7 machine rebooting during Backup & Restore

Over the last year my computer has been randomly(?) rebooting; I did a clean install of Windows 7 twice, checked temperatures, hard-drive diagnostics, did CPU stress tests & swapped out all the components except for the motherboard & harddrive to no avail. I had not experienced the issue for the last 2 months and forgot about it...


But then just today it happened again & found out how to reproduce this reboot on purpose during normal use.


I was setting up a first backup through the Backup & Restore menu and when I got to the last step in the wizard and clicked on "Backup now", the machine restarted itself.


When I go back into the Backup & Restore menu:
Menu
and click on "Manage Space" the machine reboots. I've done this 4 times now and it reboots every time. (Note this is the first time I set up this Backup scheduler, and it's off again now) I have looked through the Event viewer for anything suspicious but I didn't find anything.


What could be causing this? What actually happens during the Manage Backup and Backup now processes? Is there a way to turn on more verbose logging?


*EDIT: I also turned off "Automatically Restart" in Startup & Recovery settings, issue persists.

Data recovery on an Iomega portable drive


For Christmas, my little brother got an Iomega 500GB portable hard drive. It'd been working well, but last week it flat died, and the company's trying to shirk it, claiming it's not under warranty and saying it'll cost at least $900 to recover the data from the drive. He's still trying to fight the warranty thing, but wants to know, should it boil down to it, what other options exist for recovering the data from the drive.


(in before "BACK UP!")


Answer



Have you ascertained whether it's the drive that's actually died, or just the enclosure?


If it's the latter and the company won't admit it's under warranty you could take the drive out and either put it into another enclosure or connect it directly to a computer and see if you can recover the data that way. Don't do this while there's a chance you might get something from the company as dismantling the drive will invalidate the warranty.


If it is the drive that's failed you are looking at serious money to recover the data from it.


windows 7 - Filename too long - IMPOSSIBLE to delete/move/rename



I can`t delete a folder because it has LOTS of folders inside it. Believe me, it is really lots.


(E:\Backups\john\Users\john\AppData\Local\Google\Chrome\USERDA~1\Default\PLUGIN~1\GOOGLE~1\MAILGO~1.COM\HTTPS_~1\GOOGLE~1\41\67\34\0\69\24\78\58\62\64\5\45\81\27\61\91\95\42\27\36\91\4\2\53\92\82\21\16\18\95\47\26\71\38\69\12\67\99\35\94)


The path is even longer...


I have tried several ways for deleting it: "Unlocker Tool", "rd /s" in Command Prompt, "subst x: e:\backups\john" in Command Prompt, no one was effective.


It's IMPOSSIBLE to rename/move files.


I don't want to break up anything, so please think through your answer.


How can I get rid of it?


Answer



Try the rm utility in my Hamilton C shell. I wrote it using the new long Unicode filename format interally, so if the problem is just that the name is too long, this should get it. It sounds like this is a one-time thing, so the free demo version is all you need. If you don't want it afterward, it uninstalls cleanly in seconds.


If you try it and it doesn't work, please tell me as I'd like to fix it so it will.


email - Postfix Localhost Server Sending Spam to Itself



I have an iRedMail server that runs Postfix and am trying to determine the source of its spam problem.



I have since changed the MX records on the domain for the address receiving spam and the good news is no spam is coming through to this second server. However, new spam messages keep appearing at the old server, despite the MX record change. How is that possible?



Here is a header from one of the spam emails:



Content-Type: multipart/alternative; boundary="6656864_13052705_6656864"
Mime-Version: 1.0

Return-Path:
Content-Transfer-Encoding: 8bit
X-Virus-Scanned: Debian amavisd-new at myserverdomain.com
Received: from localhost (localhost [127.0.0.1]) by myserverdomain.com (Postfix) with ESMTP id 293FD6B977 for ; Tue, 12 Jan 2016 20:45:14 -0700 (MST)
Received: from myserverdomain.com ([127.0.0.1]) by localhost (myserverdomain.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id G3TC68wvWWEK for ; Tue, 12 Jan 2016 20:45:13 -0700 (MST)
Received: from yuijdd.stablecheck.party (unknown [46.166.133.21]) by myserverdomain.com (Postfix) with ESMTP id 480626B976 for ; Tue, 12 Jan 2016 20:45:12 -0700 (MST)
Delivered-To: nick@emaildomain.com
Message-Id:
Re: Macy's is giving out a $50 voucher to start 2016



Does this appear to be coming from my own server? Or is it coming from this IP?: 46.166.133.21. That IP is blacklisted on Spamhaus, which I have postfix set to query.



Here part of my /var/logs/mail.log file



Jan 13 03:45:11 mailhost postfix/smtpd[4796]: connect from unknown[46.166.133.21]
Jan 13 03:45:12 mailhost postfix/smtpd[4796]: 480626B976: client=unknown[46.166.133.21]
Jan 13 03:45:12 mailhost postfix/cleanup[4806]: 480626B976: message-id=
Jan 13 03:45:13 mailhost postfix/qmgr[9274]: 480626B976: from=, size=5484, nrcpt=1 (queue active)
Jan 13 03:45:13 mailhost postfix/smtpd[4796]: disconnect from unknown[46.166.133.21]

Jan 13 03:45:14 mailhost postfix/smtpd[4815]: connect from localhost[127.0.0.1]
Jan 13 03:45:14 mailhost postfix/smtpd[4815]: 293FD6B977: client=localhost[127.0.0.1]
Jan 13 03:45:14 mailhost postfix/cleanup[4806]: 293FD6B977: message-id=
Jan 13 03:45:14 mailhost postfix/qmgr[9274]: 293FD6B977: from=, size=5943, nrcpt=1 (queue active)
Jan 13 03:45:14 mailhost postfix/smtpd[4815]: disconnect from localhost[127.0.0.1]
Jan 13 03:45:14 mailhost amavis[31884]: (31884-08) Passed CLEAN, LOCAL [46.166.133.21] [46.166.133.21] -> , Message-ID: , mail_id: G3TC68wvWWEK, Hits: -0.546, size: 5482, queued_as: 293FD6B977, 568 ms
Jan 13 03:45:14 mailhost postfix/smtp[4812]: 480626B976: to=, relay=127.0.0.1[127.0.0.1]:10024, delay=2, delays=1.4/0.01/0.01/0.58, dsn=2.0.0, status=sent (250 2.0.0 from MTA([127.0.0.1]:10025): 250 2.0.0 Ok: queued as 293FD6B977)
Jan 13 03:45:14 mailhost postfix/qmgr[9274]: 480626B976: removed
Jan 13 03:45:14 mailhost postfix/pipe[4816]: 293FD6B977: to=, relay=dovecot, delay=0.09, delays=0.01/0.01/0/0.07, dsn=2.0.0, status=sent (delivered via dovecot service)



I also thought there might be a PHP script somewhere sending this mail, so I installed and ran Linux Malware Detect, but it found nothing.



Thanks for your help!



EDIT: I have blacklist filtering on in main.cf



smtpd_recipient_restrictions = 
reject_rbl_client zen.spamhaus.org,
reject_rbl_client bl.spamcop.net,

reject_rbl_client cbl.abuseat.org,
reject_rbl_client b.barracudacentral.org,
reject_invalid_hostname,
reject_unknown_sender_domain,
reject_unknown_recipient_domain,
check_policy_service inet:127.0.0.1:7777,
check_policy_service inet:127.0.0.1:10031,
reject_non_fqdn_sender,
reject_non_fqdn_hostname,
reject_non_fqdn_recipient,

reject_unlisted_recipient,
permit_mynetworks,
permit_sasl_authenticated,
reject_unauth_destination,
permit

Answer



This email came from outside your server, from the indicated IP address.



Postfix is configured to pass it to Amavis for virus scanning, and when that is complete, Amavis passes it back to Postfix for delivery.




This is why there are two localhost Received: lines. One is added by Amavis when it processes the message, and the other one when Postfix re-accepts it from Amavis for final delivery.


Windows 10 update won't install printer driver - Error 0x80240017

Recently I got a driver update for my HP printer delivered through the Windows 10 updater. It seems that the driver was successfully downloaded but cannot be properly installed, every time I try I'll get the following error:



There were problems installing some updates, but we'll try again later. If you keep seeing this and want to search the web or contact support for information, this may help:



  • Hewlett-Packard driver update for HP LaserJet CP 1025 - Error 0x80240017



The printer is actually a network printer. I checked the driver through the device manager and the newest driver is supposedly already installed. So I went ahead and uninstalled the printer (which is now a problem since I cannot simply add the printer again because I have no access to the network currently) to get rid of the update error, but the error still persists. Troubleshooting the updater also has no effect.


Now I wanted to check the registry entries (as was suggested here) for the printer in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Print but the HP printer doesn't show up any more (maybe uninstalling was too early) and therefore I have no idea what registries I could delete without breaking anything.


Does anyone have an idea on how to proceed? Maybe even a complete different approach?


EDIT:



  • the suggested approach in the comments to install the HP driver suite didn't work for me

Better control over Windows 10 update restarts

I have Windows 10 Home, normally have many things in progress, and rarely reboot. When Windows does an update and wants to restart, the notification is a very subtle "1" added to the action center icon (which I don't usually notice), and then the next morning it has shut down everything and rebooted. I know that I can't completely control the updates like I could in Windows 7, but I would at least like a clearer announcement, like a popup that stays visible until I close it. I prefer to close my programs myself and reboot on my own terms, rather than Windows doing it to me when I least expect. The question "How to stop Windows 10 from automatic restarts" shows a screenshot of a setting called "Notify to schedule restart", but I can't find such a thing on mine - my Advanced page looks like this:
Update & security -> Update settings -> Advanced options




Thoughts?



NOTE: My question is not a duplicate of How to *disable* automatic reboots in Windows 10? - that question and answer are only for Win10 Pro/Enterprise, not Win10 Home. I am even in communication with the author of that answer via his blog (who is refreshingly thorough with his testing and explanations but doesn't have access to Home), but the registry hack that is suggested there and elsewhere for Home users does not work - I have tested setting the value to 2, 3, and 5. Perhaps it used to work before the Anniversary Update, but...

ubuntu - hdparm doesn't read SSD? HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device



I'm trying to use hdparm to do secure erase of my 512GB SSD SK Hynix.
SSD has Windows 10 installed in it.



I boot Ubuntu from a usb flash drive.



sudo fdisk -l



I get a lot of different disks, turns out /dev/sda is my 8gb flash drive, not SSD.




SSD is listed as:



Disk /dev/nvme0n1: 477 GiB, 512110190592 bytes, 1000215216 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt



Now when I try to run: hdparm -I /dev/nvme0n1 terminal gives me error:
"/dev/nvme0n1:
HDIO_DRIVE_CMD(identify) failed: Inappropriate ioctl for device"



how do I use hdparm secure erase option on my SSD? From suggestion in another page on the internet, nvme-cli package was suggested, but I don't understand how that helps.



using shred command isn't recommended for SSD because SSD has limited read/write cycles, and it works very differently than HDD.



Update: my SSD is an NVMe device, not ATA, therefore hdparm command isn't suitable, use nvme -format instead.


Answer




To erase an NVMe device using nvme-cli, run nvme format -s1 .




nvme-cli package was suggested, but I don't understand how that helps.




hdparm works exclusively with ATA devices, and your SSD is an NVMe device. That doesn't mean just a new connection type (like SATA) or the same 'disk' concepts in different terms (like SCSI/SAS) – instead it's a whole different kind of storage device entirely.



None of the ATA commands apply to NVMe (the general idea of "secure erase" is present, but it's implemented differently), and so nvme-cli was suggested to you because hdparm is simply the wrong tool for the job.




(At some point Linux used to support basic SCSI-NVMe command translation, but eventually removed it as the systems are just too different (unlike SCSI-ATA translation which is successfully used). Not that it would have helped you either way, as the features accessed by hdparm are very ATA-specific and aren't covered by the translation anyway. Even basic hdparm -i doesn't work on SCSI disks.)


windows - How do I open a second Skype instance with a Batch file?



I want to launch a 2nd Skype instance from a Batch file.

I have done that successfully with the run window:



"C:\Program Files (x86)\Skype\Phone\skype.exe" /secondary


But for the life of me I cannot figure out how to do the same from a Batch file. Here is a list of my failed attempts:




  1. Nothing happens here. Command window opens and stays open:




    start "C:\Program Files (x86)\Skype\Phone\skype.exe /secondary"


  2. Command window opens and closes:



    start "C:\Program Files (x86)\Skype\Phone\skype.exe" /secondary


  3. Error message "Secondary does not exist"



    start "C:\Program Files (x86)\Skype\Phone\skype.exe" "/secondary"




What am I doing wrong?



Answer



Don't use the "start" command.



"C:\Program Files (x86)\Skype\Phone\skype.exe" /secondary


----- edit -----



or




start "Some Title" "C:\Program Files (x86)\Skype\Phone\skype.exe" /secondary


The title is required.


Sunday, August 30, 2015

windows 7 - how to properly set environment variables


I've recently started using Windows (having used Ubuntu up until now) and I find myself unable to properly set environment variables. Whenever I set them they don't seem to work. I've been going to Start->Edit Environment Variables for your Account and editing the PATH value in the upper half of the GUI. Here's what I've got so far.


 ;C:\Chocolatey\bin;C:\tools\mysql\current\bin;C:\Program Files (x86)\Git\bin;C:\Program Files\MySQL\MySQL Server 5.6\bin\;C:\Python33\Scripts;

These are each the parent directories of the executables I'd like to be able to run by name from CMD, but mysql, git, and pip aren't being recognized. Am I doing something wrong syntactically or at a general understanding level? I'd like to be able to run these commands without having to specify the full path to the executables every time.


EDIT: The full PATH extracted from CMD


PATH=C:\Python33\;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\AMD APP\bin\x86_64;C:\Program Files (x86)\AMD APP\bin\x86;C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program Files (x86)\GTK2-Runtime\bin;C:\Program Files\WIDCOMM\Bluetooth Software\;C:\Program Files\WIDCOMM\Bluetooth Software\syswow64;C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\;C:\Program Files\Microsoft SQL Server\110\Tools\Binn\;C:\Program Files (x86)\QuickTime\QTSystem\;C:\Program Files (x86)\Common Files\Acronis\SnapAPI\;C:\Program Files (x86)\Java\jre7\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\Microsoft SDKs\TypeScript\;C:\Program Files (x86)\MySQL\MySQL Utilities 1.3.4\; ;C:\Chocolatey\bin;C:\tools\mysql\current\bin

I'm being forced to use Windows by my work environment, I don't enjoy the state of affairs.


Answer



Here are some amendments i'd make. So try this command


set PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Python33;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\AMD APP\bin\x86_64;C:\Program Files (x86)\AMD APP\bin\x86;C:\Program Files\Common Files\Microsoft Shared\Windows Live;C:\Program Files (x86)\Common Files\Microsoft Shared\Windows Live;C:\Windows\System32\WindowsPowerShell\v1.0;C:\Program Files (x86)\GTK2-Runtime\bin;C:\Program Files\WIDCOMM\Bluetooth Software;C:\Program Files\WIDCOMM\Bluetooth Software\syswow64;C:\Program Files (x86)\ATI Technologies\ATI.ACE\Core-Static;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0;C:\Program Files\Microsoft SQL Server\110\Tools\Binn;C:\Program Files (x86)\QuickTime\QTSystem;C:\Program Files (x86)\Common Files\Acronis\SnapAPI;C:\Program Files (x86)\Java\jre7\bin;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit;C:\Program Files (x86)\Microsoft SDKs\TypeScript;C:\Program Files (x86)\MySQL\MySQL Utilities 1.3.4;C:\Chocolatey\bin;C:\tools\mysql\current\bin

I edited the path in notepad


PATH=C:\Python33\; <-- may not be wrong but i'd remove the \ so C:\Python33;


and these three should really be at the beginning
C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;


C:\Windows\System32\WindowsPowerShell\v1.0\; (may not be wrong but i'd remove the \ )


C:\Program Files\WIDCOMM\Bluetooth Software\; ditto


;C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\; ditto


C:\Program Files\Microsoft SQL Server\110\Tools\Binn\; <-- Binn, hmm, turns out that's right Binn. Though i'd still change \; to ;


C:\Program Files (x86)\QuickTime\QTSystem\; <-- the \


C:\Program Files (x86)\Common Files\Acronis\SnapAPI\; <-- the \


C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\; <-- the \
C:\Program Files (x86)\Microsoft SDKs\TypeScript\; <--- the \


C:\Program Files (x86)\MySQL\MySQL Utilities 1.3.4\;; <--- two semi-colons!!! that should be one semi-colon


So, try that set path=........ command I mention. If it works then you can make it permanent in the GUI. I'd do it for the system path variable and remove the user path variable. (I think the user one appends to the system one, it's unnecessaru unless you want multiple users each with their own slightly different path)


And if it fails.. then you can remove chunks and see if it works, thus troubleshooting it.


added-
If the system path is greyed out, try making the user path blank, (put it in a text file first as a backup). Then click OK, open a cmd prompt window, and now you should try appending a directory to the path within the cmd prompt and see if it works.. So,
SET PATH=%PATH%;C:\Program Files (x86)\Git\bin


now type PATH see that the above command did append Git to the Path. And see if Git runs. That tells you that the PATH as it was, has no error in it. That means the System path is OK.. (as the user path was blank so the path there is just the system path). Try not to write any directories in the user path that duplicate what is in the system path, though probably no harm if you do. Then, look at that long set PATH= that I wrote which was an amendment of yours, and execute it at the cmd prompt and see if it works. If it does, then put it in the GUI. If it doesn't then troubleshoot why, by halving it, removing Git, appending Git, see if Git runs. See, if the final directory listed in the Path is searched, then the whole Path is fine. Hopefully that method will enable you to troubleshoot it and fix the Path.


command line - Unable to delete registry key in HKLMMicrosoftWindowsCurrentVersionRun via commandline


I'm trying to delete some registry keys (startup items) via the commandline and while I can delete from HKCU\SOFTWARE\Microsoft\Windows\CurrentVersion\Run just fine I cannot delete anything from HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run.


I even created some temporary keys there through regedit.exe, but even those are undeletable via the commandline.
So I searched through similar threads here and obviously googled it. Usually this happens because:



  1. Command prompt is not run as administrator. That's not the case here as I definitely have administrator rights within the launched cmd window.

  2. Registry permissions are blocking deletion. Again, this is not the case here. I manually changed permission through regedit.exe and I'm now the owner of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run along with full control for my account.


The specific error message given by commandline is:


ERROR: The system was unable to find the specified registry key or value.

Yet the key definitely exists in the location where I'm trying to delete it from and even regedit shows its there. I can also delete it with regedit without a problem.


Anyone have any clues as what to check?


OK, so while writing this post I stumbled upon something really bizarre. It would appear that there are two versions of HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run.


I have no idea hows this even possible, but when I copy the key name in regedit they are both identical and yet they display different items. The first one is OK and it seems reg.exe is trying to find and delete the key in this location and that's why it errors out. The second one is the problematic one where the key exists that I need to delete via the commandline.


However when I add an identical key to HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Run it gets written to the "correct" location, and I can also delete it via commandline.


OK, I think I know what the problem is now. The "correct" location is displayed when I run a native 64-bit version of regedit, and the problematic one is only displayed when I run a 32-bit version of regedit.


SOLUTION:


Use %windir%\SysWOW64\reg.exe to manipulate 32-bit registry entries and %windir%\System32\reg.exe to manipulate 64-bit registry entries. I'm going to have a stern talk with the program authors though about why they shove their startup key into the 32-bit reqistry on a 64-bit system.


Answer



SOLUTION: Use %windir%\SysWOW64\reg.exe to manipulate 32-bit registry entries and %windir%\System32\reg.exe to manipulate 64-bit registry entries.


I'm going to have a stern talk with the program authors though about why they shove their startup key into the 32-bit reqistry on a 64-bit system.


amazon ec2 - Internet access from private VPC subnet?



My brain is a little bit fried trying to figure this out but I can't seem to get private subnets to connect to the internet to grab updates for example. Basically all I want is web access for these servers. I have tried solutions found in other posts but could easily be missing something since I have been looking at this for quite awhile.



So far I have done the following set up:




  • Disable source/destination checking on my NAT instance


  • Create a route table for public subnet and set its source to IGW (verified this works)

  • Create a route table for private subnet and set its source to the NAT instance

  • Create security group for NAT instance with inbound and outbound rules for 80/443 and ssh

  • Create security group for private instance with inbound and outbound 80/443 and ssh



I am able to jump on the NAT instance via SSH and then pivot onto the private server, so I know that much is working. I have verified my rules by pinging internal addresses but when I leave the local network that is when I start having problems.



Where do I need to look or what do I need to troubleshoot to help get this narrowed down? Is there some configuration that needs to be done on the NAT instance that I have overlooked?




EDIT: Configuration update




  • VPC: 172.16.0.0./16

  • Private subnet: 172.16.1.0/24

  • Public subnet: 172.16.0.0/24



Private subnet routing table:




Points to NAT instance in public subnet. Unable to access internet, attached EIP to public internet facing interface.



private vpc



Public subnet routing table:



Points the public subnet at the IGW, I have verified internet access.



enter image description here




Thoughts:



I'm starting to think there is something that needs to be set up or created on the NAT instance that lives in the public subnet to forward traffic from private instances, but I'm not exactly sure what needs to added. I found this post that explains adding a masquerade rule in iptables, is that something that apply to my situation?


Answer



yes, you must have iptables masquerading set up on the nat host.



iptables -A POSTROUTING -t nat -s (private-subnet/cidr) -j MASQUERADE



we have the same set up, you've done everything else correctly (disable source check, point private subnet to the interface of the nat instance, point public subnet to the igw).


Windows 7 failed on installation at 98%

I hope you can help me after umpteen failed attempts to load Windows 7 on an ASUS P5WD2 premium board and dual core intel 3.4Ghz 2Gig Ram processor ran the beta versions beautifully, the mini dump info is as folllows, fails at approx 98 % load on install same every time no matter what i do, have tried disabling ethernet driver no difference:


Dump File : 121810-30718-01.dmp
Crash Time : 18/12/2010 16:28:32
Bug Check String : IRQL_NOT_LESS_OR_EQUAL
Bug Check Code : 0x0000000a
Parameter 1 : 0x00000000
Parameter 2 : 0x000000ff
Parameter 3 : 0x00000000
Parameter 4 : 0x82878550
Caused By Driver : ndis.sys
Caused By Address : ndis.sys+35023
File Description : NDIS 6.20 driver
Product Name : Microsoft® Windows® Operating System
Company : Microsoft Corporation
File Version : 6.1.7600.16385 (win7_rtm.090713-1255)
Processor : 32-bit
Computer Name :
Full Path : H:\Windows\Minidump\121810-30718-01.dmp
Processors Count : 2
Major Version : 15
Minor Version : 7600


======================


Please help!


Additional Notes from Answer:


first i did have the release candidate 64 bit working fine until it ran out of time , I then tried dubious copies of 64 bit W7 which a friend guaranteed worked on his computer but the same thing happened again and again, failure , the computer unexpectedly closed down that is what it says , after up dating registry values I now have a legit copy of 32 bit W7 failure at this same point and the mini dump report of the blue screen is as described, it fails at exactly the same point in the same way every time , I have tried with no partitions on a good 500gb West Digit caviar , 2 gb Ram and an intel 3.6 ghz dual core( not the extreme ie with out hyper threading).


used a program me to extract mini dump info also in the crash report it highlights


file names : ndis.sys +35023
ntoskrnl.exe +54c11
yk62(times)86sys 12b6b


the last two do not appear in the mini dump report


I do not know where to go from here, it suggests it is a driver or some drivers that are at fault ?
But I am not sure how to solve this one am a novice at this level.

ubuntu - SSH: Unable to login with key after disabling password authentication

I have an ubuntu server is on Amazon EC2. Earlier I used to login with the original pem key file generated at instance launch time.



ssh -i /path/to/key.pem ubuntu@instance-dns-name.com


I tried to stop password authentication and made the following changes in sshd_config.



ChallengeResponseAuthentication no

PasswordAuthentication no
UsePAM no


Boom. The key login stopped working and gives me 'Permission Denied(publicKey)' error.



The funny thing is, I had created a rsa key for a non-root user using ssh-keygen. That works even now. But the ec2 pem key file will not work.



Note: If I change UsePam to 'yes', the pem key starts functioning again. I'm able to reproduce this problem on another machine. However, on the first machine I've lost the ability to gain root privileges because of key file not working. So cannot make any changes to sshd_config file.




The ssh log looks like follows.



OpenSSH_5.9p1 Debian-5ubuntu1.1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to ec2-xx.xx.us-west-2.compute.amazonaws.com [xx.xx.xx.xx] port 22.
debug1: Connection established.
debug3: Incorrect RSA1 identifier
debug3: Could not load "/home/amitoj/.ssh/backupshard.pem" as a RSA1 public key

debug1: identity file /home/amitoj/.ssh/backupshard.pem type -1
debug1: identity file /home/amitoj/.ssh/backupshard.pem-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1Debian- 5ubuntu1.1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1.1
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "ec2-xx.xx.xx.xx.us- west-2.compute.amazonaws.com" from file "/home/amitoj/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/amitoj/.ssh/known_hosts:70
debug3: load_hostkeys: loaded 1 keys

debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256- cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh- sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-rsa-cert-v01@openssh.com,ssh-dss-certv01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndaelcbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib

debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdhsha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96

debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-256-96,hmac-sha2-512,hmac-sha2-512-96,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5

debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 2d:19:d0:63:49:d9:f7:30:26:3f:6e:ff:8e:63:4d:c5
debug3: load_hostkeys: loading entries for host "ec2-xx.xx.xx.uswest-2.compute.amazonaws.com" from file "/home/amitoj/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/amitoj/.ssh/known_hosts:70
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "xx.xx.xx.xx" from file "/home/amit/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/amitoj/.ssh/known_hosts:71
debug3: load_hostkeys: loaded 1 keys

debug1: Host 'ec2-xx.xx.xx.xx.us-west-2.compute.amazonaws.com' is known and matches the ECDSA host key.
debug1: Found key in /home/amitoj/.ssh/known_hosts:70
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server

debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/amitoj/.ssh/backupshard.pem ((nil))
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey
debug1: Trying private key: /home/amitoj/.ssh/backupshard.pem
debug1: read PEM private key done: type RSA
debug3: sign_and_send_pubkey: RSA 69:ff:93:01:27:57:00:23:6e:2b:90:72:16:15:65:bb
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).



Any idea on how to gain entry to root user with the 'pem' key file?

Saturday, August 29, 2015

ubuntu - How to set default file permissions?

I have ran these codes:



sudo chown -R /
sudo chmod 755 /


And now my server is not working. I need to get backup copy of MySQL database
How can I fix it?

2 slot external hard drive dock with 1 eSATA connection?


Does anyone have experiencing using a dual slot external hard drive docking station with only one eSATA port on the back? Do both drives show up in the OS even though only one eSATA cable is connected? I'm curious because there's a dock I'm looking at that has 2 eSATA ports and many others that only have one. I'm thinking that the dock male connectors internally plug into "two" eSATA host and then are redirected to 1 single output but I want to be sure.


Answer



The result is going to depend a lot on the hardware in the dock and on your motherboard. If the dock and your eSATA controller support port multipliers, you will have no issues at all.


If not, then you will not have plug & play capabilities on both drives. You couldnt add a 2nd drive to the dock later, as the OS wouldnt see it. They would both have to be there when the device is initialized. And if both were and you unmounted one, you wouldnt be able to remount it.


Im not sure, what would happen if only one side supported port multipliers...


Can I upgrade from Windows 7 32 bit to Windows 7 64 bit without losing data?





I want to play Fallout 4 and I found out I need 64bit system in order to play it, but I have 32bit system and I'm capable of upgrading.



Can I reinstall Windows without losing pictures and that stuff if I put them in D:\files or something like that?



I don't want to buy more storage just to save my files.




I also don't want to use Google drive or similar because I have a lot of pictures on my PC.


Answer



Can I upgrade from Windows 7 32 bit to Windows 7 64 bit without losing data?




If you want to move from a 32-bit version of Windows to a 64-bit
version of Windows 7 or vice versa, you'll need to back up your
files
and choose the Custom option during Windows 7 installation.
Then, you'll need to restore your files and reinstall your programs.

For more information about performing a custom installation, see
Installing and reinstalling Windows 7‍.



Notes



To install a 64-bit version of Windows 7 on a computer running a
32-bit version of Windows, you'll need to start, or boot, your
computer using a 64-bit Windows 7 installation disc or files.



If you start your computer using a 64-bit Windows 7 installation disc

or files, but your computer isn't capable of running a 64-bit version
of Windows, you'll see a Windows Boot Manager error. You'll need to
use a 32-bit Windows 7 installation disc or files instead.




Source 32-bit and 64-bit Windows: frequently asked questions


linux - Deleting files doesn't free space



I've got a disk formatted as ext3, which was filled completely. I'm attempting to free up some space on it by deleting files, but it's not working. I can rm the files, and they don't show up in the directory listing, but I don't have free space available.



Below is a copy of attempting to delete some files. As you can see from the ls -la, the files aren't hardlinked to another location, and the delete appears to succeed. In the df output, the number of used blocks decreased by 182556, which is the space taken by the files, but the available count remained at zero. fsck detected no problems with the filesystem, and didn't change the free space at all.




I'm using CentOS 6 right now to attempt to delete the files, but most of the files were written using a Debian distribution (I'm not entirely sure which version, whatever Clonezilla uses).




# ls -la
total 182564
drwxr-xr-x. 2 root root 4096 Mar 4 2011 .
drwxr-xr-x. 5 root root 4096 Aug 13 13:18 ..
-rw-------. 1 root root 4030114 Mar 4 2011 sda1.vfat-ptcl-img.gz.aa
-rw-------. 1 root root 182667379 Mar 4 2011 sda2.ntfs-ptcl-img.gz.aa
-rw-r--r--. 1 root root 37 Mar 4 2011 sda-chs.sf

-rw-r--r--. 1 root root 31744 Mar 4 2011 sda-hidden-data-after-mbr
-rw-r--r--. 1 root root 512 Mar 4 2011 sda-mbr
-rw-r--r--. 1 root root 375 Mar 4 2011 sda-pt.parted
-rw-r--r--. 1 root root 259 Mar 4 2011 sda-pt.sf

# df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb2 484009516 473901232 0 100% /media/Images

# rm -f *


# ls -la
total 8
drwxr-xr-x. 2 root root 4096 Aug 13 15:05 .
drwxr-xr-x. 5 root root 4096 Aug 13 13:18 ..

# df .
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb2 484009516 473718676 0 100% /media/Images


# cd /media ; umount Images

# e2fsck -fv /dev/sdb2
e2fsck 1.41.12 (17-May-2010)
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information


589 inodes used (0.00%)
37 non-contiguous files (6.3%)
0 non-contiguous directories (0.0%)
# of inodes with ind/dind/tind blocks: 278/256/0
119390761 blocks used (97.89%)
0 bad blocks
5 large files

551 regular files
29 directories

0 character device files
0 block device files
0 fifos
0 links
0 symbolic links (0 fast symbolic links)
0 sockets
--------
580 files

# mount /dev/sdb2 Images


# df Images
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sdb2 484009516 473718676 0 100% /media/Images

Answer



The ext3 and ext4 family of filesystems reserve a portion of the disk to keep it from becoming truly "100% full", for stability reasons, and so that the FS can store metadata in some of that space.



You can directly tweak how much of the space is reserved:




tune2fs -m 1 /dev/sdXX


replacing "1" with the percentage (0 to 100) of the disk to reserve, and "XX" with the device node and partition number of the device node, so in your case "XX" would be "b2".



Analogy: you have a full bus, and the only remaining seat is next to a very large person who occupies a seat and a half by himself. A person who is very insistent on getting a seat walks up and demands to sit there. Although most people would consider the second seat taken, this person is insistent. So the large person goes, "Whoa, OK!" and squishes in to allow them to sit. But as soon as 1 person gets off the bus and opens up another seat, even if the passenger next to the large person moves into a seat, most people still consider the bus to be 100% full, because nobody wants to sit next to the large person.



Source



You can also check this to see Reserved Blocks count…




dumpe2fs -h /dev/sdb2


To quote the very intelligent user who nailed the issue in the source above:




You will see "Available" go positive when
"Used" is reduced to below 0.95*136236548 blocks = 129424720 blocks
roughly.





(we have to adjust the "0.95" to the reserved % in your specific case, and the 136236548 blocks to the total block size of your device).


linux - Bash: amount of bytes used in a log file grouped by token


Assume a large log file of several GBs and several million lines where each line contains a token identifying the user account that generated the line.


All tokens have the same length and can be found at the position within each log line.


The goal is to figure out the amount of bytes logged by each account.


One way of doing this is in multiple steps, like this:


awk -F "|" '{ print $5 }' trace.log | sort | uniq | xargs -l sh -c 'echo -n $0 && grep "$0" trace.log | wc -c'

where awk extracts the token (5th entry tokenizing by '|'), sort | uniq extracts the list of unique tokens appearing in the file and finally xargs greps and counts the bytes.


Now this works but it is terribly inefficient because the same (huge) file gets grepped X times.


Is there a smarter way of achieving the same via shell commands? (where by smarter I mean faster and without consuming tons of RAM or temporary storage, like sorting the whole file in RAM or sorting it to a tmp file).


Answer



Try:


awk -F "|" '{ a[$5]+=1+length($0) } END{for (name in a) print name,a[name]}' trace.log

Example


Let's consider this test file:


$ cat trace.log
1|2|3|4|jerry|6
a|b|c|d|phil|f
1|2|3|4|jerry|6

The original command produces this output:


$ awk -F "|" '{ print $5 }' trace.log | sort | uniq | xargs -l sh -c 'echo -n $0 && grep "$0" trace.log | wc -c'
jerry32
phil15

The suggested command, which loops through the file just once, produces this output:


$ awk -F "|" '{ a[$5]+=1+length($0) } END{for (name in a) print name,a[name]}' trace.log
jerry 32
phil 15

How it works



  • -F "|"


    This sets the field separator for input.


  • a[$5]+=1+length($0)


    For each line, we add the length of the line to the count stored in associative array a under this line's user name.


    The quantity length($0) does not include the newline that ends the line. Consequently, we add one to this to account for the \n.


  • END{for (name in a) print name,a[name]}


    After we have read through the file once, we print out the sums.



windows 7 - delete hidden folder/file with name exceeding 256 characters


I ended up with a folder that cannot be deleted.
The issue started out as Windows 7 complained about a deletion I performed in an old recyclebin with a message 'filename too long 434 instead of 256 characters' (or something very similar).
I tried lots of things like those mentioned in this very similar question: Renaming/deleting impossible filenames in Windows 7


But I did not succeed.
I succeeded in renaming and removing lots of subfolders but not all of them.
This is what's left visibly, a folder from the recyclebin:


C:\RECYCLER\S-1-5-21-842925246-2025429265-682008880-1013\

Inside this folder is another one named com4 but that is only visible to a tool like Total-Commander. I was not able to display this folder com4 by say:


dir *.* /A:s
dir *.* /A:h

I find interesting that apparently something is in there because the following command results in: access denied. So Windows 7 recognizes an object but does not reveal or delete it.


rd "\\c:\RECYCLER\S-1-5-21-842925246-2025429265-682008880-1013\com4\

Even though I am using an elevated command prompt and logged in as admin without any limitations (I assume).


Any ideas how to get rid of this folder named com4?
Getting rid of anything below c:\RECYCLER is fine as well.


Answer



To deal with the too long of a path/filename you can shorten the names of the folders until the path/file gets short enough to delete.


Even so it looks like you have another problem. Windows can't delete some files names that get created(com4 in your case). You can try deleting these from Linux, or possibly some software other than Windows Explorer to delete it.



In addition, in Windows and DOS utilities, some words are also
reserved and cannot be used as filenames.[17] For example, DOS device
files:[19]


CON, PRN, AUX, CLOCK$, NUL COM1, COM2, COM3, COM4 LPT1, LPT2, LPT3,
LPT4 (LPT4 only in some versions of DR-DOS) LST (only in 86-DOS and
DOS 1.xx) KEYBD$, SCREEN$ (only in multitasking MS-DOS 4.0) $IDLE$
(only in Concurrent DOS 386, Multiuser DOS and DR DOS 5.0 and higher)
CONFIG$ (only in MS-DOS 7.0-8.0)



https://en.wikipedia.org/wiki/Filename


Note: I've seen some similar problems that can get fixed by using the 8.3/MSDOS versions of the filenames.



dir /X (that's a capital X.)



This command will show the the filenames in 8.3/MSDOS.


There's more information about this on the page linked to in the description:
Renaming/deleting impossible filenames in Windows 7


linux - How to SSH to ec2 instance in VPC private subnet via NAT server

I have created a VPC in aws with a public subnet and a private subnet. The private subnet does not have direct access to external network. S...