A Technical Reference Guide to Coin Flips and Dice Rolls

Index

1. Overview

A BIP39 mnemonic phrase (commonly called a seed phrase) is a human-readable representation of cryptographic entropy. Instead of memorizing 256 random bits, you memorize a short list of ordinary words that encode exactly those bits.

The pipeline that turns physical randomness into a mnemonic is:

Physical randomness
        ↓
Binary entropy
        ↓
Checksum (SHA-256)
        ↓
Entropy + checksum
        ↓
Split into 11-bit groups
        ↓
BIP39 word lookup
        ↓
Mnemonic phrase

The physical randomness can come from:

• Coin flips
• Dice rolls
• Hardware random number generators

The single most important requirement is:

The initial entropy must be unpredictable and unbiased.

Everything else in this guide is bookkeeping. If the entropy is weak, no amount of correct encoding will save the wallet.

Why bother with coins and dice at all?

A hardware wallet’s built-in RNG is almost certainly fine. The reason to do this by hand is that “almost certainly” is a claim you cannot verify: a compromised or subtly broken generator produces output that looks perfectly random to you while being entirely predictable to whoever broke it. This has happened in the field more than once.

Physical randomness moves the trust from the device to a process you can watch. It does not make the wallet more random — 256 bits is 256 bits — it makes the randomness auditable by you.

2. BIP39 Entropy Sizes

BIP39 defines a fixed set of entropy sizes. The entropy must be a multiple of 32 bits, between 128 and 256 bits, and each size pairs with a checksum whose length is entropy_bits / 32.

The checksum length follows directly from the entropy length:

checksum_bits = entropy_bits / 32

For example, 128 / 32 = 4 and 256 / 32 = 8.

12 or 24 words? 128 bits is already far beyond brute force, and no realistic advance changes that. 24 words buys margin against implementation mistakes and partial backup loss, not against an attacker with a bigger computer. Both are fine; 24 costs you roughly twice the physical work described below.

3. The BIP39 Word List

The BIP39 English dictionary contains exactly 2048 words, and that number is not arbitrary:

2048 = 2^11

Because there are 2^11 words, each word encodes exactly 11 bits. That is why the total bit count always divides evenly by 11:

• 132 bits / 11 = 12 words
• 264 bits / 11 = 24 words

Two properties of the list are worth knowing before you start writing words down:

• The first four letters are unique. No two words in the English list share their first four letters, so abandon can be recorded as aban without ambiguity. Hardware wallets rely on this when they let you type only four characters.
• The language is part of the encoding. The same index means a different word in the Spanish, French, or Japanese list. A phrase is only recoverable if you know which word list produced it, so unless you have a reason otherwise, use English — it is what every wallet supports.

Use the official list from the BIP39 specification. A “similar” list found elsewhere will silently produce a different wallet.

4. Method 1 — Generating Entropy with a Coin

4.1 Why a Coin Works

A fair coin has two equally likely outcomes, which maps perfectly onto a single binary digit:

Each flip therefore provides exactly 1 bit of entropy — clean, unbiased, and with no conversion math required. This directness is what makes the coin the easiest source to reason about.

4.2 How Many Flips You Need

Since each flip is one bit, the number of flips equals the entropy size:

A sequence such as H T H H T T H T ... becomes the binary string 1 0 1 1 0 0 1 0 ....

4.3 If You Do Not Trust Your Coin

Real coins are not perfectly fair. A flipped coin caught in the hand lands the same side up it started roughly 50.8% of the time — a physical wobble effect predicted by Diaconis and later confirmed over 350,757 recorded flips. Spinning a coin on a table is far worse, and can be biased by several percent.

A 50.8% bias costs you almost nothing — across all 256 flips it removes less than a twentieth of one bit (256 × H(0.508) ≈ 255.95). Two habits shrink it further at no cost: flip properly and let the coin land on the floor rather than catching it, and vary which face starts upward, since the effect is a same-side bias rather than a heads bias. If you would rather not rely on any of that, two fixes remove the bias entirely:

Von Neumann extraction. Flip in pairs and discard the ties:

Because HT and TH have identical probability for any fixed bias, the output bits are perfectly uniform. The cost is throughput: at best you keep one bit per four flips, so a 24-word seed needs around 1024 flips on average. Correct, and tedious.

Hash the flips. Record all 256 flips as a string and run it through SHA-256, exactly as the dice method does in §7.2. This removes bias without discarding anything, and it is what most tooling does in practice.

Either way, the rule is the same: do not correct a suspected bias by hand. Adjusting your own results reintroduces exactly the human non-randomness you were trying to avoid.

5. Coin Flip Conversion Process

Step 1 — Record the flips

Using Heads = 1 and Tails = 0, write down every flip in order:

1011010011011001...

This binary sequence is your entropy.

Step 2 — Verify the length

Count the bits and confirm they match the target exactly — 128 bits for a 12-word mnemonic, 256 bits for a 24-word mnemonic. Do not pad, trim, or “round off” the sequence.

Writing the bits in groups of eight makes the count checkable at a glance: 16 groups for 128 bits, 32 groups for 256 bits.

Step 3 — Calculate the checksum

Compute the SHA-256 hash of the raw entropy bytes (not the text of the bit string):

checksum = first (entropy_bits / 32) bits of SHA-256(entropy)

• 128-bit entropy → take the first 4 bits of the hash.
• 256-bit entropy → take the first 8 bits of the hash.

This is the one step you cannot do in your head. Hashing "1011..." as ASCII text instead of as bytes is the single most common mistake here, and it produces a phrase that is valid-looking but wrong.

Step 4 — Append the checksum

Concatenate the checksum bits onto the end of the entropy:

Entropy   : 1011010011011001…   (128 or 256 bits)
Checksum  : 0110                 (4 or 8 bits)
Combined  : 1011010011011001…0110

The result is 132 bits (12 words) or 264 bits (24 words). (The short strings above are illustrative — real entropy is 128 or 256 bits long.)

Step 5 — Split into 11-bit groups

Chop the combined sequence into groups of 11 bits:

10110100110  →  group 1
11001010101  →  group 2
...

Each 11-bit group is a number in the range 0 (00000000000) to 2047 (11111111111).

Step 6 — Map numbers to words

Each number indexes into the BIP39 word list:

• Index 0 → the first word (abandon)
• Index 2047 → the last word (zoo)

Read the words out in order and you have your mnemonic.

Doing steps 3–6 offline

Steps 3 through 6 are mechanical, and doing them by hand is error-prone. On an air-gapped machine you will wipe afterwards, the whole conversion is a few lines:

import hashlib

bits = "1011010011011001..."          # your recorded flips, 128 or 256 of them
assert len(bits) in (128, 160, 192, 224, 256)

entropy  = int(bits, 2).to_bytes(len(bits) // 8, "big")
digest   = hashlib.sha256(entropy).digest()
checksum = format(int.from_bytes(digest, "big"), "0256b")[: len(bits) // 32]
combined = bits + checksum

indices = [int(combined[i:i + 11], 2) for i in range(0, len(combined), 11)]
print(indices)                        # look these up in the BIP39 English word list

Never run this — or any BIP39 tool — on a machine that is online, and never paste your bits into a website, however reputable it looks.

6. Method 2 — Generating Entropy with Dice

6.1 Dice Entropy

A six-sided die has 6 equally likely outcomes, so each roll provides:

log2(6) ≈ 2.585 bits

That is more than double a coin flip, so you need far fewer physical actions. Round up, so that the rolls are guaranteed to carry at least the required number of bits:

(128 / 2.585 ≈ 49.5 → 50, and 256 / 2.585 ≈ 99.0 → 100.)

Checking it the other way round confirms the margin: 50 rolls produce 6⁵⁰ ≈ 2¹²⁹ equally likely sequences, and 100 rolls produce 6¹⁰⁰ ≈ 2²⁵⁸·⁵ — comfortably above 128 and 256 bits respectively.

6.2 Recording the Rolls

Map each face to a base-6 digit (0–5) so the results can be treated as one large number:

A roll sequence of 3 6 2 1 5 4 becomes the digits 2 5 1 0 4 3, i.e. the base-6 number 251043₆.

7. Converting Dice Rolls to Entropy

There are two practical approaches. The second is what most hardware wallets actually use.

7.1 Base-6 Conversion (conceptual)

Interpret the whole roll sequence as one large base-6 number and convert it to binary. For the small example 251043₆:

2×6⁵ + 5×6⁴ + 1×6³ + 0×6² + 4×6¹ + 3
  = 15552 + 6480 + 216 + 0 + 24 + 3
  = 22275  (decimal)
  = 101011100000011  (binary)

This illustrates the idea, but naïvely truncating a base-6 number to a fixed bit length can introduce slight bias, because 6ⁿ is never a power of two.

7.2 SHA-256 of the Rolls (recommended)

The clean, unbiased, and widely used method is to hash the recorded rolls:

entropy = SHA-256("digits from every roll, concatenated")

Roll ~50 or ~100 times, write the digits as a single string, hash it with SHA-256, and use the resulting bits as your entropy. Because SHA-256 output is computationally indistinguishable from uniform, this sidesteps base-conversion bias entirely. This is the approach used by tools such as Coldcard and SeedSigner.

Two details matter:

• For a 12-word seed, take the first 128 bits of the 256-bit digest. For 24 words, use all 256 bits.
• Hashing does not create entropy. SHA-256 always emits 256 bits, but those bits carry only as much randomness as went in. Fifty D6 rolls hashed into a 24-word phrase gives you a 24-word phrase with roughly 129 bits of real security, not 256. If you want 256 bits, you must roll ~100 times. The output looks identical either way, which is what makes this mistake dangerous.

8. Warning: Why Naïve Dice Encoding Is Biased

A die has six states, and six is not a power of two:

6 = 2 × 3     →     6 ≠ 2ⁿ

This means you cannot just assign each face a 3-bit pattern:

1 = 000   2 = 001   3 = 010
4 = 011   5 = 100   6 = 101

Three bits can represent 8 values, but a die only produces 6. The patterns 110 and 111 never appear, so this mapping wastes states and skews the bit distribution: four of the six faces begin with 0, so the leading bit is 0 two-thirds of the time instead of half. Across a whole seed that is a large, systematic loss — an attacker who knows you used this mapping searches a much smaller space than the word count suggests.

Use an unbiased method instead:

• SHA-256 of the rolls (recommended)
• Base-6 → binary conversion done carefully
• Rejection sampling — read two rolls as a number 0–35, discard 32–35, and keep the remaining 32 outcomes as a clean 5 bits
• Dice whose face count is a power of two — a D8 gives a clean 3 bits and a D16 a clean 4 bits, which is exactly what Method 3 below exploits

9. Method 3 — D8 / D16 / D16 Dice (Direct Word Indexing)

9.1 Why This Combination Works

The whole problem with a D6 is that 6 is not a power of two. That problem disappears entirely when every die is a power of two:

And:

3 + 4 + 4 = 11 bits  →  exactly one BIP39 word

So one D8 and two D16 dice, rolled together, select one word directly. No base conversion, no SHA-256 over the rolls, no rejection sampling, no wasted states. Every outcome maps to exactly one of the 2048 words, and every word is equally likely.

This is the method documented by Blockstream for generating a recovery phrase offline: https://help.blockstream.com/generate-recovery-phrase-offline-d8-d16-d16

9.2 The Dice

Label the three dice and always read them in the same order:

Use three visually distinct dice (different colours) so you can throw all three at once and still read them unambiguously. Order matters — swapping Second and Third gives a different word.

9.3 The Index Formula

index = (First − 1) × 2⁸  +  (Second − 1) × 2⁴  +  (Third − 1)
      = (First − 1) × 256 +  (Second − 1) × 16  +  (Third − 1)

The range is exactly right, with nothing left over:

minimum:  (1,1,1)     →  0×256 + 0×16 + 0   =    0    (abandon)
maximum:  (8,16,16)   →  7×256 + 15×16 + 15 = 2047    (zoo)

Dice numbered from zero. Some D16 (hex) dice are labelled 0–F and some D8 are labelled 0–7. If a die already starts at 0, use its face value directly and drop the − 1 for that die.

Seen as bits, the three dice simply fill the 11-bit word index from left to right:

First  (D8)  → 3 bits   b10 b9 b8
Second (D16) → 4 bits   b7 b6 b5 b4
Third  (D16) → 4 bits   b3 b2 b1 b0

Worked example — you roll First = 6, Second = 13, Third = 2:

(6 − 1) × 256  +  (13 − 1) × 16  +  (2 − 1)
= 1280 + 192 + 1
= 1473

binary:  101  1100  0001   →   10111000001

Index 1473 is the 1474th word in the BIP39 list, because index 0 is the first word (abandon).

9.4 How Many Rolls You Need

Compare that with 256 coin flips or ~100 D6 rolls plus a hand-verified SHA-256. Each set of three is also independently checkable: if you mis-record one word, only that word is wrong, and the rest of the phrase is unaffected.

9.5 The Last Word Is Not Free

This is the one catch, and it is easy to get wrong.

Because this method produces words directly, it bypasses the “entropy → checksum → split” pipeline. Words 1…11 (or 1…23) are pure entropy and can be taken exactly as rolled. The final word cannot, because part of it is the checksum:

So roll the last set normally, then keep only the leading bits of that provisional word and let the checksum determine the rest:

• 12 words — keep the first 7 bits of the provisional 12th word. 2⁴ = 16 words share that prefix; exactly one of them has a valid checksum.
• 24 words — keep the first 3 bits of the provisional 24th word. 2⁸ = 256 words share that prefix; exactly one of them has a valid checksum.

Those leading bits are not decoration — they are real entropy, and they are what completes the count:

12 words:  11 words × 11 bits + 7 bits = 121 + 7 = 128 bits
24 words:  23 words × 11 bits + 3 bits = 253 + 3 = 256 bits

Use an offline, trusted BIP39 tool or hardware wallet to work out which candidate is valid. Most hardware wallets do this for you: you type the first 11 (or 23) words and the device only offers valid final words — pick the one matching your rolled prefix.

If you would rather compute it yourself, again on an air-gapped machine:

import hashlib

indices = [ ... ]     # the 11 or 23 words you rolled, as indices 0–2047
last    = 1473        # the provisional final word, exactly as rolled

n_ent  = 128 if len(indices) == 11 else 256
n_keep = n_ent - len(indices) * 11        # 7 bits (12 words) or 3 bits (24 words)
n_free = 11 - n_keep
prefix = format(last, "011b")[:n_keep]

for tail in range(2 ** n_free):
    cand = int(prefix + format(tail, "0" + str(n_free) + "b"), 2)
    bits = "".join(format(i, "011b") for i in indices + [cand])
    ent  = int(bits[:n_ent], 2).to_bytes(n_ent // 8, "big")
    cs   = format(int.from_bytes(hashlib.sha256(ent).digest(), "big"), "0256b")[: n_ent // 32]
    if bits[n_ent:] == cs:
        print("valid final index:", cand)

Do not just re-roll until the last word happens to be valid. It is technically unbiased (2⁷ = 128 of the 2048 words are valid last words for a 12-word phrase, 2³ = 8 for a 24-word phrase), but it needs an average of 16 re-rolls for 12 words and 256 for 24 words — and you still need a checksum tool to know when to stop.

9.6 Why This Method Is Worth the Extra Dice

10. Complete BIP39 Pipeline

Coin Method

256 coin flips
        ↓
256-bit entropy
        ↓
SHA-256  →  8 checksum bits
        ↓
264-bit sequence
        ↓
24 groups of 11 bits
        ↓
24 BIP39 words

Dice Method

~100 dice rolls
        ↓
SHA-256(rolls)  →  256-bit entropy
        ↓
SHA-256  →  8 checksum bits
        ↓
264-bit sequence
        ↓
24 groups of 11 bits
        ↓
24 BIP39 words

D8/D16/D16 Method

24 sets of (D8, D16, D16)
        ↓
index = (D8−1)×256 + (D16−1)×16 + (D16−1)
        ↓
24 word indices, 0–2047
        ↓
words 1–23 kept as rolled
        ↓
word 24: keep first 3 bits,
last 8 bits = SHA-256 checksum
        ↓
24 BIP39 words

11. Security Comparison

All three are secure when done correctly. The coin trades speed for simplicity; the D6 trades simplicity for speed; the D8/D16/D16 set buys both back at the cost of needing dice you cannot pick up in any shop.

12. From Mnemonic to Wallet Seed

The words are not the wallet. BIP39 defines one more step, which turns the mnemonic into the 512-bit seed that BIP32 uses to derive every key and address:

mnemonic words  +  optional passphrase
        ↓
PBKDF2-HMAC-SHA512
password = mnemonic sentence
salt     = "mnemonic" + passphrase
2048 iterations, 64-byte output
        ↓
512-bit seed
        ↓
BIP32 master key  →  accounts  →  addresses

Three consequences are worth understanding before you rely on a passphrase:

• The passphrase is not checksummed. The mnemonic is; the passphrase is not. Any passphrase is “valid”, so a single typo silently derives a completely different, empty wallet — with no error message. If you use one, back it up as carefully as the words, and separately from them.
• Every passphrase gives a real wallet. This is what makes hidden or decoy wallets possible: the same 24 words plus different passphrases are different wallets, and nothing in the backup reveals how many exist.
• The passphrase is not a substitute for entropy. 2048 PBKDF2 iterations is a very light stretching by modern standards. A guessable passphrase over a stolen mnemonic is brute-forceable; it protects a lost backup only if it is itself high-entropy.

If you skip the passphrase, the salt is simply the string "mnemonic" — which is the default every wallet uses.

13. The Role of the Checksum

A valid BIP39 phrase carries built-in error detection. If a single word is wrong, the checksum almost always fails and the wallet rejects the phrase as invalid.

“Almost always” is literal: a 24-word phrase has 8 checksum bits, so a random corruption still passes by chance about 1/256 of the time, and a 12-word phrase (4 bits) about 1/16 of the time. The checksum catches careless mistakes; it is not a guarantee.

It is important to understand what the checksum is not: it is not a security feature. It provides only:

• Typo detection
• Phrase validation

The real security comes entirely from the quality of the random entropy. A checksum on predictable entropy protects nothing.

14. Verify Before You Fund the Wallet

A seed phrase you have never restored from is a backup you have never tested. Before sending anything of value:

1. Check the phrase validates. Enter it into your offline tool or hardware wallet and confirm the checksum is accepted.
2. Wipe and restore. Reset the device and recover it from the words as you wrote them down — not from what is still on screen. This is the step that catches a mis-copied word, a wrong order, or an illegible character in your backup.
3. Confirm the same first address. After restoring, the first receive address must match what you saw before the wipe. If it does not, check the derivation path before panicking — BIP44 (m/44'/0'/0'), BIP49, BIP84 (m/84'/0'/0', native SegWit) and BIP86 (Taproot) produce entirely different addresses from the same seed.
4. Cross-check with a second implementation. Deriving the first address with an independent offline tool confirms you are not depending on one vendor’s bug.
5. Send a small test amount first, and only then the rest.

If you used a passphrase, repeat the whole restore with it — an untested passphrase is the most common way people lock themselves out permanently.

15. Best Practices

Use a fair physical source

Good:

• A quality, balanced coin
• Casino-grade dice
• Any genuinely random physical process

Avoid:

• Human-chosen numbers
• Recognizable patterns
• Dates, phone numbers, or “keyboard mashing”

Humans are notoriously bad at being random. The whole point of a physical source is to remove your brain from the loop.

Keep entropy offline

Generate the entropy:

• Offline, on a trusted air-gapped device
• Away from cameras and onlookers
• With no network access

Never use a website to convert your flips or rolls into words, even one that claims to work offline — you cannot verify what the page you actually loaded is doing. The same applies to phone photos, cloud notes, password managers that sync, and printers with storage.

Never reuse entropy

Every wallet must have its own unique randomness. Do not:

• Reuse a seed across wallets
• Modify or “improve” existing words by hand
• Change words to make them more memorable

Altering a valid phrase by hand almost always breaks the checksum, and in the rare cases it does not, you have quietly reduced your entropy to something an attacker can search.

Record the backup properly

• Write the words numbered and in order — order is part of the secret.
• Prefer a durable medium (stamped steel) over paper for anything long-lived.
• Note the word list language and the derivation path alongside the backup; neither is secret, and both save you later.

16. Summary

The entire process, end to end:

Physical randomness
        ↓
Entropy extraction
        ↓
SHA-256 checksum
        ↓
Append checksum
        ↓
Split into 11-bit numbers
        ↓
Map numbers 0–2047 to words
        ↓
BIP39 mnemonic phrase
        ↓
PBKDF2 (+ optional passphrase)
        ↓
512-bit wallet seed

A coin yields randomness directly as binary — simple, slow, and hard to get wrong. A D6 yields more entropy per action but requires careful, unbiased conversion (ideally by hashing the rolls with SHA-256). A D8 plus two D16 skips the conversion entirely: three throws = 11 bits = one word, with the only special case being the checksum-bearing final word.

Performed correctly, all three produce a mnemonic that is every bit as secure as one from a hardware RNG — with the added assurance that you watched the randomness happen.

Overview

A block is the basic building unit of a blockchain, holding a group of verified transactions along with cryptographic references to preceding blocks. Together, they create an unchangeable ledger within the blockchain network.

In essence, each block serves as a digital container that securely and permanently records transaction information for the entire system.

A block in Bitcoin is like a page in a ledger that records and secures recent transactions.

Each block is linked to the previous one, forming the blockchain.

Structure of a Bitcoin Block

A Bitcoin block has two main parts The Block Header and The Block Body

Block Header

The metadata of the block, containing:

• Version → Protocol version used to create the block.
• Previous Block Hash → Cryptographic hash of the previous block, linking the chain.
• Merkle Root → A hash representing all transactions in the block, built via a Merkle Tree.
• Timestamp → Approximate creation time of the block.
• Difficulty Target (nBits) → The target value that the block hash must meet (controls mining difficulty).
• Nonce → A 32-bit number miners vary to find a valid block hash.

Block Body

• Contains the list of transactions included in the block.

• Starts with a coinbase transaction:

  • The first transaction in every block.
  • Rewards the miner with new Bitcoin (block subsidy + transaction fees).

Block Constraints

Block Size & Limits

• Current average block size: ~1–2 MB (due to SegWit optimization).
• Can hold a few thousand transactions depending on size.

Block Reward

• New bitcoins are created via the block reward in the coinbase transaction.
• Halves every 210,000 blocks (~4 years).
• Started at 50 BTC (2009) → now 6.25 BTC (until 2024 halving → 3.125 BTC).

Block Time

• Target: 10 minutes per block.
• Achieved by adjusting difficulty every 2016 blocks (~2 weeks) so the average time stays near 10 minutes.

Block Validation

• A valid block must:

  • Have a hash below the difficulty target (proof-of-work).
  • Contain valid, non-duplicate transactions.
  • Correctly reference the previous block.
  • Correctly compute the Merkle root.

Blockchain Linking

• Each block references the hash of the previous block.

• This makes the blockchain immutable:

  • Changing one block invalidates all subsequent blocks (since the hashes break).

Block Header

At the top of every Bitcoin block is an 80-byte (160 hex character) block header that summarizes all the data in the block. This header is crucial for:

• Block identification and validation
• Creating the block hash
• Maintaining blockchain integrity through cryptographic linking

Miners repeatedly hash this block header to try and get a result below the current target. Successfully finding such a hash allows the block to be added to the blockchain, a process known as mining.

Block Header Structure

The 80-byte header contains six key fields:

Retrieving Block Header Data

To get the block header for block height 1 using Blockchain.com’s API:

curl 'https://blockchain.info/rawblock/1?format=hex' | head -c 160 > block-1.txt

010000006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1cdb606e857233e0e61bc6649ffff001d01e36299

Decoding the Block Header

Here’s a Python function to decode a block header from hex format:

import struct
import binascii
import datetime

def decode_block_header(hex_header):
    header_bytes = bytes.fromhex(hex_header)
    
    version = struct.unpack('<I', header_bytes[0:4])[0]
    prev_block = header_bytes[4:36][::-1].hex()
    merkle_root = header_bytes[36:68][::-1].hex()
    timestamp = struct.unpack('<I', header_bytes[68:72])[0]
    bits = struct.unpack('<I', header_bytes[72:76])[0]
    nonce = struct.unpack('<I', header_bytes[76:80])[0]

    return {
        'version': version,
        'previous_block_hash': prev_block,
        'merkle_root': merkle_root,
        'timestamp': f"{timestamp} ({datetime.fromtimestamp(timestamp, UTC).strftime('%Y-%m-%d %H:%M:%S %Z')})",
        'bits': bits,
        'nonce': nonce
    }

#-----------------------------------------------------------------------------------------------------------
def main():
    # Example usage
    hex_header = '010000006fe28c0ab6f1b372c1a6a246ae63f74f931e8365e15a089c68d6190000000000982051fd1e4ba744bbbe680e1fee14677ba1a3c3540bf7b1cdb606e857233e0e61bc6649ffff001d01e36299'
    decoded = decode_block_header(hex_header)

    for key, value in decoded.items():
        print(f"{key}: {value}")

#-----------------------------------------------------------------------------------------------------------
if __name__ == "__main__":
    main()

Example Output

version: 1
previous_block_hash: 000000000019d6689c085ae165831e934ff763ae46a2a6c172b3f1b60a8ce26f
merkle_root: 0e3e2357e806b6cdb1f70b54c3a3a17b6714ee1f0e68bebb44a74b1efd512098
timestamp: 1231469665 (2009-01-09 02:54:25)
bits: 486604799
nonce: 2573394689

Key Notes

• All integer fields are stored in little-endian format
• Hash fields (Previous Block Hash and Merkle Root) need byte reversal for proper display
• The timestamp represents when mining began, not necessarily when the block was found

Block Body

The block body contains the transactions that the block commits to. It immediately follows the 80-byte block header.

The block body (unlike the header) is variable length and encodes all spending activity included by miners in that block.

The block body is where all the actual bitcoin transfers happen - the header just contains metadata and proof-of-work, while the body contains the economic activity.

Block Body Structure

Transaction Serialization (legacy)

[version: 4 bytes LE]
[input_count: varint]
  for each input:
    [prev_txid: 32 bytes (little-endian in storage)]
    [prev_index: 4 bytes LE]
    [scriptSig_length: varint]
    [scriptSig: scriptSig_length bytes]
    [sequence: 4 bytes LE]
[output_count: varint]
  for each output:
    [value: 8 bytes LE]        # satoshis
    [pk_script_length: varint]
    [pk_script: pk_script_length bytes]
[locktime: 4 bytes LE]

Notes

• prev_txid is stored in byte-order used within the block (usually little-endian in raw bytes); when displayed by explorers the bytes are reversed to big-endian hex.
• value is an unsigned 8-byte little-endian integer of satoshis.
• sequence historically used for RBF / timelocks; typically 0xffffffff.

Transaction Serialization (SegWit - BIP141)

[version: 4 bytes LE]
[marker: 1 byte == 0x00]
[flag:   1 byte == 0x01]
[input_count: varint]
  for each input: (same fields as legacy *but* scriptSig is usually small or empty)
[output_count: varint]
  for each output: (same as legacy)
[witnesses for each input]
[locktime: 4 bytes LE]

Notes

• SegWit transactions add a 2-byte header after version
• for each stack element: [witness_item_len: varint] [witness_item: bytes]
• The presence of SegWit is detected when marker == 0x00 and flag != 0x00 immediately after the version.

Retrieving Block Body Data

To get the block body for block height 1 using Blockchain.com’s API:

curl 'https://blockchain.info/rawblock/1?format=hex' | tail -c +161 > block-1-body.txt

0101000000010000000000000000000000000000000000000000000000000000000000000000ffffffff0704ffff001d0104ffffffff0100f2052a0100000043410496b538e853519c726a2c91e61ec11600ae1390813a627c66fb8be7947be63c52da7589379515d4e0a604f8141781e62294721166bf621e73a82cbf2342c858eeac00000000

Decoding the genesis block.

import json
import struct
import requests
from typing import TypedDict


class BlockHeader(TypedDict):
    """Type definition for decoded block header data."""

    version: int
    previous_block_hash: str
    merkle_root: str
    timestamp: int
    bits: int
    nonce: int


class TransactionInput(TypedDict):
    """Type definition for decoded block tx input data."""

    previous_tx_hash: str
    previous_output_index: int
    script_sig: str
    script_sig_length: str
    sequence: str
    is_coinbase: bool


class TransactionOutput(TypedDict):
    """Type definition for decoded block transaction output data."""

    value_satoshi: int
    value_btc: float
    script_pubkey: str
    script_pubkey_length: int


class WitnessItem(TypedDict):
    """Type definition for a single witness item."""
    data: str
    length: int
    type: str
    description: str


class TransactionWitness(TypedDict):
    """Type definition for witness data of a single input."""
    input_index: int
    witness_count: int
    witness_items: list[WitnessItem]
    witness_type: str
    total_witness_size: int


class Transaction(TypedDict):
    version: int
    tx_type: str
    has_witness: bool
    tx_input_count: int
    tx_inputs: list[TransactionInput]
    tx_output_count: int
    tx_outputs: list[TransactionOutput]
    lock_time: int
    size_bytes: int
    total_output_value_satoshi: int
    total_output_value_btc: float
    tx_witnesses: list[TransactionWitness]


class BlockDecoder:
    """
    A comprehensive Bitcoin block decoder that handles both Legacy and SegWit transactions.
    Provides methods to decode block headers, block bodies, and complete blocks.
    """

    def __init__(self):
        """Initialize the Bitcoin block decoder."""
        pass

    # --------------------------------------------------------------------------------------------
    def decode_full_block(self, block):
        """
        """
        try:
            # First 80 bytes (160 hex characters) are the header
            if len(block) < 160:
                raise ValueError("Block data too short to contain a valid header")

            header = block[:160]
            body = block[160:]

            decoded_header = self.decode_block_header(header)
            decoded_body = self.decode_block_body(body)

            return {
                'block_hash': self.calculate_block_hash(header),
                'header': decoded_header,
                'body': decoded_body,
                'total_block_size_bytes': len(block) // 2,
                'header_size_bytes': 80,
                'body_size_bytes': len(body) // 2
            }

        except Exception as ex:
            raise ValueError({
                'error': f"Failed to decode full block: {str(ex)}",
                'raw_hex_length': len(block)
            })

    # --------------------------------------------------------------------------------------------
    @staticmethod
    def decode_block_header(header: str) -> BlockHeader:
        """

        """
        # Validate input
        match header:
            case str() if header.strip():
                header = header.strip().lower()
            case _:
                raise ValueError("Header must be a non-empty string")

        # Validate hex string length (80 bytes = 160 hex characters)
        if len(header) != 160:
            raise ValueError(f"Header must be exactly 160 hex characters, got {len(header)}")

        # Validate hex string format and convert to bytes
        try:
            header_bytes = bytes.fromhex(header)
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        try:
            (version,) = struct.unpack("<I", header_bytes[0:4])
            previous_block_hash = header_bytes[4:36][::-1].hex()
            merkle_root = header_bytes[36:68][::-1].hex()
            (timestamp,) = struct.unpack("<I", header_bytes[68:72])
            (bits,) = struct.unpack("<I", header_bytes[72:76])
            (nonce,) = struct.unpack("<I", header_bytes[76:80])
        except ValueError as e:
            raise struct.error(f"Failed to unpack header data: {e}") from e

        block_header: BlockHeader = {
            "version": version,
            "previous_block_hash": previous_block_hash,
            "merkle_root": merkle_root,
            "timestamp": timestamp,
            "bits": bits,
            "nonce": nonce,
        }
        return block_header

    # --------------------------------------------------------------------------------------------
    @staticmethod
    def calculate_block_hash(header) -> str:
        """
        Calculate the block hash from the header.

        Args:
            header (str): Block header as hex string

        Returns:
            str: Block hash (double SHA-256, reversed)
        """
        try:
            import hashlib
            header_bytes = bytes.fromhex(header)
            hash1 = hashlib.sha256(header_bytes).digest()
            hash2 = hashlib.sha256(hash1).digest()
            return hash2[::-1].hex()  # Reverse byte order
        except Exception as ex:
            raise ValueError(f"Error calculating hash: {str(ex)}")

    # --------------------------------------------------------------------------------------------
    def decode_block_body(self, body: str):
        """
        """

        # Validate data
        match body:
            case str() if body.strip():
                body = body.strip().lower()
            case _:
                raise ValueError("Block body data must be a non-empty string")

        # Validate hex string format and convert to bytes
        try:
            body_bytes: bytes = bytes.fromhex(body)
        except ValueError as ex:
            raise ValueError(f"Invalid hex string: {ex}") from ex

        try:
            offset: int = 0
            transactions = []
            total_inputs = 0
            total_outputs = 0
            total_value_satoshi = 0
            has_segwit_txs = False
            coinbase_count = 0

            # Transaction count (varint)
            tx_count, offset = self.decode_varint(body_bytes, offset)

            # Extract and retrieve the list of raw hex txs.
            raw_txs: list[str] = self.extract_raw_txs(body_bytes.hex())

            for raw_tx in raw_txs:
                try:
                    tx: Transaction = self.decode_raw_tx(raw_tx)
                    transactions.append(tx)
                    total_inputs += tx['tx_input_count']
                    total_outputs += tx['tx_output_count']
                    total_value_satoshi += tx['total_output_value_satoshi']

                    # Check if any transaction is SegWit
                    if tx['has_witness']:
                        has_segwit_txs = True

                    # Count coinbase transactions
                    if any(inp['is_coinbase'] for inp in tx['tx_inputs']):
                        coinbase_count += 1

                except Exception as ex:
                    raise Exception(f"Error decoding raw_tx {raw_tx}: {ex}") from ex

            return {
                'transaction_count': tx_count,
                'transactions_decoded': len([tx for tx in transactions if 'error' not in tx]),
                'transactions_failed': len([tx for tx in transactions if 'error' in tx]),
                'block_type': 'SegWit Block' if has_segwit_txs else 'Legacy Block',
                'has_segwit_transactions': has_segwit_txs,
                'coinbase_transactions': coinbase_count,
                'total_inputs': total_inputs,
                'total_outputs': total_outputs,
                'total_output_value_satoshis': total_value_satoshi,
                'total_output_value_btc': total_value_satoshi / 100000000.0,
                'block_body_size_bytes': len(body_bytes),
                'transactions': transactions
            }

        except Exception as ex:
            raise Exception(f"Error decoding block body {ex}") from ex

    # -------------------------------------------------------------------------------------------------
    def decode_raw_tx(self, tx: str) -> Transaction:
        """
        """

        # Validate data
        match tx:
            case str() if tx.strip():
                tx = tx.strip().lower()
            case _:
                raise ValueError("Transaction data must be a non-empty string")

        # Validate hex string format and convert to bytes
        try:
            tx_bytes: bytes = bytes.fromhex(tx)
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        try:
            offset: int = 0

            # Version (4 bytes)
            version: int = struct.unpack('<I', tx_bytes[offset:offset + 4])[0]
            offset += 4

            # Check for SegWit marker and flag
            has_witness = False
            if offset + 1 < len(tx_bytes) and tx_bytes[offset] == 0x00 and tx_bytes[offset + 1] == 0x01:
                has_witness = True
                offset += 2  # Skip marker (0x00) and flag (0x01)

            # Decode inputs
            tx_inputs: list[TransactionInput] = []
            # Input count (varint)
            tx_input_count, offset = self.decode_varint(tx_bytes, offset)
            for _ in range(tx_input_count):
                tx_inputs, offset = self.decode_transaction_inputs_from_raw_tx(tx_bytes.hex())

            # Decode outputs
            tx_outputs: list[TransactionOutput] = []
            # Output count (varint)
            tx_output_count, offset = self.decode_varint(tx_bytes, offset)
            for _ in range(tx_output_count):
                tx_outputs, offset = self.decode_transaction_outputs_from_raw_tx(tx_bytes.hex())

            # Decode witnesses
            tx_witnesses: list[TransactionWitness] = []
            if has_witness:
                # Witness count (varint)
                tx_witness_count, offset = self.decode_varint(tx_bytes, offset)
                for _ in range(tx_witness_count):
                    tx_witnesses, offset = self.decode_transaction_witnesses_from_raw_tx(tx_bytes.hex())

            # Lock time (4 bytes)
            lock_time = struct.unpack('<I', tx_bytes[offset:offset + 4])[0]
            offset += 4

            # Calculate total output value
            total_output_value = sum(tx_output['value_satoshi'] for tx_output in tx_outputs)

            decoded_tx: Transaction = {
                'version': version,
                'tx_type': 'SegWit' if has_witness else 'Legacy',
                'has_witness': has_witness,
                'tx_input_count': tx_input_count,
                'tx_inputs': tx_inputs,
                'tx_output_count': tx_output_count,
                'tx_outputs': tx_outputs,
                'lock_time': lock_time,
                'size_bytes': offset,
                'total_output_value_satoshi': total_output_value,
                'total_output_value_btc': total_output_value / 100000000.0,
                'tx_witnesses': tx_witnesses
            }

        except ValueError as e:
            raise struct.error(f"Failed to unpack transaction data: {e}") from e

        return decoded_tx

    # --------------------------------------------------------------------------------------------
    def extract_raw_txs(self, body: str, offset: int = 0) -> list[str]:
        """
        """

        # Validate data
        match body:
            case str() if body.strip():
                body = body.strip().lower()
            case _:
                raise ValueError("Block body data must be a non-empty string")

        # Validate hex string format and convert to bytes
        try:
            body_bytes: bytes = bytes.fromhex(body)
        except ValueError as ex:
            raise ValueError(f"Invalid hex string: {ex}") from ex

        txs: list[str] = []

        try:
            # read transaction count
            tx_count, offset = self.decode_varint(body_bytes, offset)

            for _ in range(tx_count):
                start_offset = offset

                # --- parse just enough to find tx end ---
                version = body_bytes[offset:offset + 4]
                offset += 4

                has_witness = False
                if body_bytes[offset] == 0x00 and body_bytes[offset + 1] == 0x01:
                    has_witness = True
                    offset += 2

                # inputs
                tx_inputs_count, offset = self.decode_varint(body_bytes, offset)

                for _ in range(tx_inputs_count):
                    offset += 32  # prev txid
                    offset += 4  # index
                    script_len, offset = self.decode_varint(body_bytes, offset)
                    offset += script_len
                    offset += 4  # sequence

                # outputs
                tx_outputs_count, offset = self.decode_varint(body_bytes, offset)
                for _ in range(tx_outputs_count):
                    offset += 8  # value
                    script_len, offset = self.decode_varint(body_bytes, offset)
                    offset += script_len

                # witnesses
                if has_witness:
                    for _ in range(tx_inputs_count):
                        n_stack, offset = self.decode_varint(body_bytes, offset)
                        for _ in range(n_stack):
                            item_len, offset = self.decode_varint(body_bytes, offset)
                            offset += item_len

                # locktime
                offset += 4

                # slice raw tx
                tx_end = offset
                txs.append(body_bytes[start_offset:tx_end].hex())
        except ValueError as ex:
            raise ValueError(f"Error extracting raw txs: {ex}") from ex

        return txs

    # --------------------------------------------------------------------------------------------
    def decode_transaction_inputs_from_raw_tx(self, tx: str, offset: int = 0) -> tuple[list[TransactionInput], int]:
        """
        """

        # Validate and convert input
        if not isinstance(tx, str) or not tx.strip():
            raise ValueError("Transaction must be a non-empty string")

        try:
            tx_bytes = bytes.fromhex(tx.strip())
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        if len(tx_bytes) < 10:  # Minimum transaction size
            raise ValueError("Transaction too short to be valid")

        try:
            # Skip version (4 bytes)
            offset += 4

            # Check for SegWit marker and flag
            has_witness = False
            if offset + 1 < len(tx_bytes) and tx_bytes[offset] == 0x00 and tx_bytes[offset + 1] == 0x01:
                has_witness = True
                offset += 2  # Skip marker (0x00) and flag (0x01)

            # Input count
            tx_inputs_count, offset = self.decode_varint(tx_bytes, offset)

            tx_inputs: list[TransactionInput] = []

            for _ in range(tx_inputs_count):
                prev_tx_hash = tx_bytes[offset:offset + 32][::-1].hex()
                offset += 32

                prev_output_index = struct.unpack("<I", tx_bytes[offset:offset + 4])[0]
                offset += 4

                script_len, offset = self.decode_varint(tx_bytes, offset)

                script_sig = tx_bytes[offset:offset + script_len].hex()
                offset += script_len

                sequence = struct.unpack("<I", tx_bytes[offset:offset + 4])[0]
                offset += 4

                is_coinbase = (prev_tx_hash == "00" * 32 and prev_output_index == 0xFFFFFFFF)

                tx_inputs.append({
                    "previous_tx_hash": prev_tx_hash,
                    "previous_output_index": prev_output_index,
                    "script_sig": script_sig,
                    "script_sig_length": hex(script_len),
                    "sequence": hex(sequence),
                    "is_coinbase": is_coinbase,
                })
        except (struct.error, IndexError) as e:
            raise ValueError(f"Failed to decode transaction: {e}") from e
        return tx_inputs, offset

    # -------------------------------------------------------------------------------------------------
    def decode_transaction_outputs_from_raw_tx(self, tx: str, offset: int = 0) -> tuple[list[TransactionOutput], int]:
        """
        """

        # Validate and convert input
        if not isinstance(tx, str) or not tx.strip():
            raise ValueError("Transaction must be a non-empty string")

        try:
            tx_bytes = bytes.fromhex(tx.strip())
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        if len(tx_bytes) < 10:  # Minimum transaction size
            raise ValueError("Transaction too short to be valid")

        try:
            # Skip version (4 bytes)
            offset += 4

            # Check for SegWit marker and flag
            has_witness = False
            if offset + 1 < len(tx_bytes) and tx_bytes[offset] == 0x00 and tx_bytes[offset + 1] == 0x01:
                has_witness = True
                offset += 2  # Skip marker (0x00) and flag (0x01)

            # Input count and skip inputs
            tx_inputs_count, offset = self.decode_varint(tx_bytes, offset)

            # Skip all inputs
            for _ in range(tx_inputs_count):
                # Skip previous tx hash (32 bytes) + output index (4 bytes)
                offset += 36

                # Skip script sig
                script_pubkey_length, offset = self.decode_varint(tx_bytes, offset)
                offset += script_pubkey_length

                # Skip sequence (4 bytes)
                offset += 4

            # Output count
            tx_outputs_count, offset = self.decode_varint(tx_bytes, offset)

            tx_outputs: list[TransactionOutput] = []

            for i in range(tx_outputs_count):
                if offset + 8 > len(tx_bytes):
                    raise ValueError(f"Insufficient data for output {i} value")

                # Value (8 bytes, little-endian)
                value_satoshi = struct.unpack("<Q", tx_bytes[offset:offset + 8])[0]
                offset += 8

                # Script length and script
                script_pubkey_length, offset = self.decode_varint(tx_bytes, offset)

                if offset + script_pubkey_length > len(tx_bytes):
                    raise ValueError(f"Insufficient data for output {i} script")

                script_pubkey = tx_bytes[offset:offset + script_pubkey_length].hex()
                offset += script_pubkey_length

                block_tx_output: TransactionOutput = {
                    "value_satoshi": value_satoshi,
                    "value_btc": value_satoshi / 100000000.0,
                    "script_pubkey": script_pubkey,
                    "script_pubkey_length": script_pubkey_length
                }

                tx_outputs.append(block_tx_output)

        except (struct.error, IndexError) as e:
            raise ValueError(f"Failed to decode transaction: {e}") from e

        return tx_outputs, offset

    def decode_transaction_witnesses_from_raw_tx(self, tx: str) -> tuple[list[TransactionWitness], int]:
        """
        """
        # Validate and convert input
        if not isinstance(tx, str) or not tx.strip():
            raise ValueError("Transaction must be a non-empty string")

        try:
            tx_bytes = bytes.fromhex(tx.strip())
        except ValueError as e:
            raise ValueError(f"Invalid hex string: {e}") from e

        if len(tx_bytes) < 10:  # Minimum transaction size
            raise ValueError("Transaction too short to be valid")

        offset = 0

        try:
            # Skip version (4 bytes)
            offset += 4

            # Check for SegWit marker + flag
            if not (offset + 1 < len(tx_bytes) and
                    tx_bytes[offset] == 0x00 and
                    tx_bytes[offset + 1] == 0x01):
                raise ValueError("Transaction is not SegWit format (no witness marker/flag)")

            offset += 2  # Skip marker and flag

            # Input count
            input_count, offset = self.decode_varint(tx_bytes, offset)

            # Skip all inputs
            for _ in range(input_count):
                # Skip previous tx hash (32 bytes) + output index (4 bytes)
                offset += 36

                # Skip script sig
                script_len, offset = self.decode_varint(tx_bytes, offset)
                offset += script_len

                # Skip sequence (4 bytes)
                offset += 4

            # Skip outputs
            output_count, offset = self.decode_varint(tx_bytes, offset)

            for _ in range(output_count):
                # Skip value (8 bytes)
                offset += 8

                # Skip script pubkey
                script_len, offset = self.decode_varint(tx_bytes, offset)
                offset += script_len

            # Now decode witness data
            witnesses: list[TransactionWitness] = []

            for input_index in range(input_count):
                # Number of witness items for this input
                witness_count, offset = self.decode_varint(tx_bytes, offset)

                witness_items: list[WitnessItem] = []
                total_witness_size = 0

                # Decode each witness item
                for item_index in range(witness_count):
                    # Length of this witness item
                    item_len, offset = self.decode_varint(tx_bytes, offset)

                    if offset + item_len > len(tx_bytes):
                        raise ValueError(f"Insufficient data for witness item {item_index} of input {input_index}")

                    # Extract witness item data
                    item_data = tx_bytes[offset:offset + item_len]
                    offset += item_len

                    # Analyze the witness item
                    witness_item = self.decode_witness_item(item_data, item_index, witness_count)
                    witness_items.append(witness_item)
                    total_witness_size += item_len

                # Determine witness type
                witness_type = self.determine_witness_type(witness_count, witness_items)

                witnesses.append(TransactionWitness(
                    input_index=input_index,
                    witness_count=witness_count,
                    witness_items=witness_items,
                    witness_type=witness_type,
                    total_witness_size=total_witness_size,
                ))

        except (struct.error, IndexError) as e:
            raise ValueError(f"Failed to decode transaction witnesses: {e}") from e

        return witnesses, offset

    # -------------------------------------------------------------------------------------------------
    def decode_witness_item(self, data: bytes, index: int, witness_count: int) -> WitnessItem:
        """
        """
        hex_data = data.hex()
        length = len(data)

        # Empty witness item
        if length == 0:
            return WitnessItem(
                data="",
                length=0,
                type="EMPTY",
                description="Empty witness item"
            )

        # For P2WPKH (2 items: signature + pubkey)
        if witness_count == 2:
            if index == 0:
                # First item should be signature (typically 70-72 bytes with DER encoding + sighash)
                if 70 <= length <= 73:
                    sighash_type = data[-1] if length > 0 else 0
                    sighash_desc = self.get_sighash_description(sighash_type)
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - {sighash_desc}"
                    )
                else:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - non-standard length"
                    )
            elif index == 1:
                # Second item should be public key (33 bytes compressed or 65 bytes uncompressed)
                if length == 33:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="PUBLIC_KEY",
                        description="Compressed public key (33 bytes)"
                    )
                elif length == 65:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="PUBLIC_KEY",
                        description="Uncompressed public key (65 bytes)"
                    )
                else:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="PUBLIC_KEY",
                        description=f"Public key ({length} bytes) - non-standard length"
                    )

        # For P2WSH (multisig or script)
        elif witness_count > 2:
            if index == 0:
                # First item is typically empty for multisig due to OP_CHECKMULTISIG bug
                if length == 0:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="EMPTY",
                        description="Empty item (OP_CHECKMULTISIG dummy)"
                    )
            elif index == witness_count - 1:
                # Last item is typically the redeem script
                return WitnessItem(
                    data=hex_data,
                    length=length,
                    type="REDEEM_SCRIPT",
                    description=f"Witness script/redeem script ({length} bytes)"
                )
            else:
                # Middle items are typically signatures
                if 70 <= length <= 73:
                    sighash_type = data[-1] if length > 0 else 0
                    sighash_desc = self.get_sighash_description(sighash_type)
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - {sighash_desc}"
                    )
                else:
                    return WitnessItem(
                        data=hex_data,
                        length=length,
                        type="SIGNATURE",
                        description=f"ECDSA signature ({length} bytes) - non-standard length"
                    )

        # Single witness item (unusual)
        elif witness_count == 1:
            return WitnessItem(
                data=hex_data,
                length=length,
                type="UNKNOWN",
                description=f"Single witness item ({length} bytes) - possibly custom script"
            )

        # Default case
        return WitnessItem(
            data=hex_data,
            length=length,
            type="UNKNOWN",
            description=f"Unknown witness data ({length} bytes)"
        )

    # -------------------------------------------------------------------------------------------------
    @staticmethod
    def determine_witness_type(witness_count: int, witness_items: list[WitnessItem], /) -> str:
        """
        Determine the overall witness type based on witness items.

        Args:
            witness_count: Number of witness items
            witness_items: List of analyzed witness items

        Returns:
            String describing the witness type
        """
        match witness_count:
            case 0:
                return "NO_WITNESS"
            case 1:
                return "CUSTOM_SCRIPT"
            case 2:
                # P2WPKH pattern: signature + pubkey
                if (len(witness_items) == 2 and
                        witness_items[0].get("type") == "SIGNATURE" and
                        witness_items[1].get("type") == "PUBLIC_KEY"):
                    return "P2WPKH"
                else:
                    return "UNKNOWN_2_ITEM"
            case count if count > 2:
                # P2WSH multisig pattern: empty + signatures + script
                if (witness_items and
                        witness_items[0].get("type") == "EMPTY" and
                        witness_items[-1].get("type") == "REDEEM_SCRIPT"):
                    sig_count = sum(1 for item in witness_items[1:-1]
                                    if item.get("type") == "SIGNATURE")
                    return f"P2WSH_MULTISIG_{sig_count}_OF_N"
                elif witness_items and witness_items[-1].get("type") == "REDEEM_SCRIPT":
                    return "P2WSH_CUSTOM_SCRIPT"
                else:
                    return f"COMPLEX_WITNESS_{count}_ITEMS"
            case _:
                return "UNKNOWN_WITNESS"

    # -------------------------------------------------------------------------------------------------
    @staticmethod
    def get_sighash_description(sighash_type: int, /) -> str:
        """
        Get description of sighash type.

        Args:
            sighash_type: Sighash type byte

        Returns:
            Human-readable description of sighash type
        """
        base_type = sighash_type & 0x1f
        anyonecanpay = bool(sighash_type & 0x80)

        match base_type:
            case 0x01:
                base_desc = "SIGHASH_ALL"
            case 0x02:
                base_desc = "SIGHASH_NONE"
            case 0x03:
                base_desc = "SIGHASH_SINGLE"
            case _:
                base_desc = f"SIGHASH_UNKNOWN({base_type:02x})"

        if anyonecanpay:
            return f"{base_desc} | SIGHASH_ANYONECANPAY"
        else:
            return base_desc

    # --------------------------------------------------------------------------------------------
    @staticmethod
    def decode_varint(data: bytes, offset: int) -> tuple[int, int]:
        """
        Decode a variable-length integer from the data starting at offset.

        Args:
            data (bytes): The byte data
            offset (int): Starting position

        Returns:
            tuple: (value, new_offset)
        """
        first_byte = data[offset]
        if first_byte < 0xFD:
            return first_byte, offset + 1
        elif first_byte == 0xFD:
            return struct.unpack("<H", data[offset + 1: offset + 3])[0], offset + 3
        elif first_byte == 0xFE:
            return struct.unpack("<I", data[offset + 1: offset + 5])[0], offset + 5
        elif first_byte == 0xFF:
            return struct.unpack("<Q", data[offset + 1: offset + 9])[0], offset + 9
        else:
            raise ValueError("Invalid varint prefix byte")


# -------------------------------------------------------------------------------------------------
def main():
    block_decoder: BlockDecoder = BlockDecoder()

    # set block height here
    block_height = 0  # change this to any block height you want

    url = f"https://blockchain.info/rawblock/{block_height}?format=hex"
    resp = requests.get(url)

    if resp.status_code == 200:
        block = resp.text.strip()  # raw block hex
        block_bytes = bytes.fromhex(block)
        decoded_block = block_decoder.decode_full_block(block_bytes.hex())
        print(json.dumps(decoded_block, indent=2))
    else:
        print("Error:", resp.status_code, resp.text)


if __name__ == "__main__":
    main()

Piensa por ti mismo para decidir.

1. Lo que quieres.
2. Lo que es verdad.
3. Lo que hay que hacer para obtener (1) a la luz de (2)

Hazlo con humildad y apertura de miras para tener en cuenta las mejores ideas de las que dispones.

Busca siempre patrones de las cosas que te afectan, para entender las relaciones causa-efecto que las impulsan y extraer principios que te sirvan para afrontarlas de forma eficaz.

1. Admite y afronta la realidad.

1.1 Sé hiperrealista.

a. Sueños + realidad + determinación = una vida de éxito.

1.2 La verdad y una comprensión acertada de la realidad es la base primordial de cualquier resultado positivo.

1.3 Ten la mente radicalmente abierta. Actúa con total transparencia.

a. La transparencia y la apertura de miras radicales son imprescindibles para aprender rápido y cambiar de verdad.

b. No dejes que el miedo al qué dirán se interponga en tu camino.

c. La sinceridad y la transparencia radical atraerán tareas y relaciones más significativas.

1.4 Observa la naturaleza para comprender el funcionamiento de la realidad.

a. No te obsesiones con tu visión de cómo deberían ser las cosas, o no aprenderás como son en realidad.

b. Para ser bueno, un hecho debe operar en consonancia con las leyes de la realidad y contribuir al desarrollo.

c. La evolución es la fuerza más poderosa del universo, la única permanente y la que lo impulsa todo.

d. Renuévate o muere.

1.5 La evolución es el mayor logro y la mayor recompensa que puede haber en la vida.

a. Los incentivos del individuo han de estar alineados con los objetivos del grupo.

b. La realidad opera en beneficio del conjunto, no del individuo.

c. La adaptación mediante el ensayo y error rápidos es de un valor incalculable.

d. Eres todo y nada a la vez. Decide que quieres ser.

e. Lo que eres depende de tu punto de vista.

1.6 Entiende las lecciones de la naturaleza.

a. Maximiza tu progreso.

b. Recuerda que sin sacrificio no hay progreso.

c. Para fortalecerse, la ley natural dicta que uno ha de exceder sus limites, y eso duele.

1.7 Dolor + reflexión = progreso

a. No rehúyas el dolor, ábrele los brazos.

b. Acepta el amor exigente.

1.8 Sopesa las consecuencias de segundo y tercer orden.

1.9 Atente a las consecuencias de tus actos.

1.10 Observa la maquinaria desde arriba.

a. Piensa en tí mismo como una máquina dentro de otra, y sé consciente de la capacidad para alterar ambas y obtener mejores resultados.

b. Comparando los resultados y los objetivos, puedes decidir cómo modificar la maquinaria.

c. Distingue entre tú como diseñador de tu maquinaria y como operario de esta.

d. El craso error de la mayoría de las personas es que no se ven a sí mismas ni a los demás de forma objetiva, y eso les lleva a tropezar repetidamente con las debilidades propias y ajenas.

e. Son triunfadores quienes pueden sobreponerse a sí mismos, para mirar las cosas con objetividad y aprovecharse de ellas a fin de propiciar un cambio.

f. Pedir ayuda a aquellos cuyas fuerzas se complementan con tus debilidades es una habilidad que ha de desarrollarse a toda costa, por cuanto implicará la construcción de barreras que te impedirán hacer lo que no debas.

g. Como es difícil observarse a uno mismo objetivamente, has de confiar en la opinión de los demás y en cualquier hecho demostrable.

h. Con una mente lo bastante abierta y decidida, puedes lograr prácticamente cualquier objetivo.

2. Utiliza el proceso de 5 pasos para obtener lo que quieras de la vida.

2.1 Ten objetivos claros.

a. Prioriza: Cuando se puede lograr prácticamente todo, es imposible conseguir nada.

b. No confundas metas con deseos.

c. Decidir lo que quieres en la vida consiste en conciliar objetivos y deseos.

d. No confundas las señales de éxito con el propio éxito.

e. No descartes un objetivo por considerarlo inalcanzable.

f. Recuerda que con un liston alto se desarrollan grandes capacidades.

g. Nada te detendrá si eres a) flexible b) autónomo.

h. Aceptar cómo lidiar con tus tropiezos es tan importante como seguir adelante.

2.2 Identifica y no tolores los problemas.

a. Considera los problemas dolorosos como mejoras en potencia que te hacen señas evidentes.

b. No evites afrontar los problemas porque provienen de realidades desagradables que no es grato afrontar.

c. Identifica los problemas de manera especifica.

d. No confundas la causa de un problema con el problema en si mismo.

e. Distingue los problemas grandes de los pequeños.

f. Una vez identificado el problema, no lo toleres.

2.3 Diagnostica los problemas para llegar a sus causas últimas.

a. Primero debes saber qué pasa y luego decidir qué vas a hacer.

b. Distingue las causas primeras de las causas últimas.

c. Admite que si conoces a alguien (tambien a ti mismo) sabrás qué esperar de él.

2.4 Traza un plan.

a. Retrocede antes de avanzar.

b. Considera los problemas como una serie de resultados producidos por una maquina.

c. Recuerda que normalmente hay muchos caminos para llegar a un mismo fin.

d. Piensa en un plan como en un guión cinematográfico que muestre qué debe hacer cada uno a los largo del tiempo.

e. Considera poner tu plan por escrito, para que todos lo vean y usen como baremo de tu progreso.

f. Reconoce que no necesitas demasiado tiempo para diseñar un buen plan.

2.5 Transforma las estrategias en resultados.

a. Los buenos planificadores que no ponen en práctica sus ideas van sin rumbo.

b. Los buenos habitos de trabajo se subestiman demasiado.

c. Establece parámetros claros para asegurarte de que sigues el plan.

2.6 Recuerda que las flaquezas no importan si les encuentras soluciones.

a. Busca patrones en los errores e identifica qué paso.

b. Cada cual tiene un obstáculo principal que se interpone entre él y él éxito; debes encontrar el tuyo y afrontarlo debidamente.

2.7 Comprende tu esquema mental y tu humildad, así como los ajenos.

3. Sé radicalmente abierto de miras.

3.1 Reconoce las dos barreras.

a. Entiende la barrera del ego.

b. Los dos yos y su lucha por el control.

c. Entiende la barrera del punto ciego.

3.2 Practica una apertura de miras radical.

a. Cree sinceramente que, quizá, deconozcas cuál es el mejor camino, y reconoce que la capacidad de aceptar esa ignorancia es mas importante que todo lo que sepas.

b. Reconoce que las decisiones se toman en dos etapas: primero se recopila toda la información relevante y luego se decide.

c. No te preocupes por la apariencias, sino por el objetivo.

d. Reconoce que es imposible producir sin aprender.

e. Reconoce que, para ponerte efectivamente en la piel del otro y ver el mundo desde su perspectiva, has de de proceder a una suspención temporal del juicio: solo desde la empatía podrás evaluar correctamente una perspectiva ajena.

f. Recuerda que buscas la mejor respuesta, no la mejor respuesta que puedas llegar por tu cuenta.

g. Deja claro si estás discutiendo o si quieres aprender y endenter. Piensa en qué opción es mejor, según tu credibilidad propia y la de los demás.

3.3 Aprecia el arte del desacuerdo reflexivo.

3.4 Triangula tu perspectiva con gente creíble dispuesta a discrepar.

a. Hacer planes para que el peor caso factible sea lo mejor posible.

3.5 Reconoce los sintomas de la cerrazón y la apertura de miras y préstales atención.

3.6 Entiende cómo se puede llegar a ser radicalmente abierto de miras.

a. Utiliza el dolor con regularidad como guía hacia unas reflexiones de calidad.

b. Haz de la apertura de miras una constumbre.

c. Conoce tus puntos ciegos.

d. Si varias personas creíbles te dicen que estas obrando mal, y eres el único que no lo ve así, da por supuesto que seguramente no estés siendo neutral.

e. Medita.

f. Básate en los hechos y anima a los demás a que hagan los mismo.

g. Haz todo lo que esté en tus manos para ayudar a los demás a ser tambien abiertos de miras.

h. Utilza una herramienta de decisión basada en los hechos.

i. Comprende cuándo es mejor rendirse y tener fe en tu sistema de toma de decisiones.

4. Entiende que las personas funcionan de formas muy distintas.

4.1 Entiende el poder que se deriva de comprender cómo funcionas tú y los demás.

a. Naces con atributos que pueden ser una ayuda o un escollo, según cómo los utilices.

4.2 El trabajo y las relaciones significativas no son solo cosas bonitas que escoges, sino que las llevas programadas genéticamente.

4.3 Entiende las grandes batallas mentales y cómo controlarlas para obtener lo que quieres.

a. Entiende que la mente consciente está enzarzada en una batalla con la mente inconsciente.

b. Sé consciente de que la luca más constante que existe es la de los sentimientos y el pensamiento.

c. Compagina los sentimientos y el pensamiento.

d. Escoge bien tus hábitos.

e. Instruye al yo inferior de forma afable y persistente para desarrollar buenos hábitos.

f. Entiende la diferencia entre el hemisferio derecho y el izquierdo, y el pensamiento derivado de cada uno.

g. Entiende lo mucho y lo poco que el cerebro puede cambiar.

[AWS] Tagging Resources

Amazon Web Services allows customers to assign metadata to their AWS resources in the form of tags. Each tag is a simple label consisting of a customer-defined key and an optional value that can make it easier to manage, search for, and filter resources by purpose, owner, environment, or other criteria. AWS tags can be used for many purposes.

Important

Do not add personally identifiable information (PII) or other confidential or sensitive information in tags. Tags are accessible to many AWS services, including billing. Tags are not intended to be used for private or sensitive data.

Best practices

As you create a tagging strategy for AWS resources, follow best practices:

• Do not store personally identifiable information (PII) or other confidential or sensitive information in tags.

• Use a standardized, case-sensitive format for tags, and apply it consistently across all resource types.

• Consider tag guidelines that support multiple purposes, like managing resource access control, cost tracking, automation, and organization.

• Use automated tools to help manage resource tags. AWS Resource Groups and the Resource Groups Tagging API enable programmatic control of tags, making it easier to automatically manage, search, and filter tags and resources.

• Use too many tags rather than too few tags.

• Remember that it is easy to change tags to accommodate changing business requirements, but consider the consequences of future changes. For example, changing access control tags means you must also update the policies that reference those tags and control access to your resources.

Best Practices for Naming Tags

Adopt a Standardized Approach for Tag Names

Keep in mind that names for AWS tags are case sensitive so ensure that they are used consistently. For example, the tags CostCenter and costcenter are different, so one might be configured as a cost allocation tag for financial analysis and reporting and the other one might not be. Similarly, the Name tag appears in the AWS Console for many resources, but the name tag does not.

A number of tags are predefined by AWS or created automatically by various AWS services. Many AWS-defined tags are named using all lowercase, with hyphens separating words in the name, and prefixes to identify the source service for the tag.

For example:

• aws:ec2spot:fleet-request-id identifies the Amazon EC2 Spot Instance Request that launched the instance

• aws:cloudformation:stack-name identifies the AWS CloudFormation stack that created the resource

• lambda-console:blueprint identifies blueprint used as a template for an AWS Lambda function

• elasticbeanstalk:environment-name identifies the application that created the resource

Important

Consider naming your tags using all lowercase, with hyphens separating words, and a prefix identifying the organization name or abbreviated name.

For example:

• anycompany:cost-center to identify the internal Cost Center code

• anycompany:environment-type to identifywhether the environment is development, test, or production

• anycompany:application-id to identify the application the resource was created for

The prefix ensures that tags are clearly identified as having been defined by your organization and not by AWS or athird-party tool that you may be using. Using all lowercase with hyphens for separators avoids confusion about how to capitalize a tag name. For example, anycompany:project-idis simpler to rememberthan ANYCOMPANY:ProjectID, anycompany:projectID, or Anycompany:ProjectId

Standardize Names for AWS Resources

[AWS] Naming Conventions Best Practices

Tagging categories

Technical Tags:

1. name: Used to give a descriptive name to the resource. For example, “web-server-prod” or “db-dev”.
2. owner: Specifies the person or team responsible for the resource. This can be an individual’s name, team name, or email address.
3. availability-zone: Specifies the availability zone or region where the resource is located.
4. service: Indicates the AWS service associated with the resource, such as “ec2”, “s3”, “rds”, etc.
5. lifecycle: Specifies the lifecycle stage of the resource, such as “active”, “inactive”, “archived”.
6. version: Indicates the version of the resource if applicable.
7. created-date: Specifies the date when the resource was created.
8. last-modified-date: Specifies the date when the resource was last modified.
9. resource-id: Unique identifier for the resource.
10. environment-type: Specifies the type of environment, such as “test”, “qa”, or “production”.

Tags for Automation:

1. automated: Indicates whether the resource is managed by automation tools. Values can be “yes” or “no”.
2. auto-scaling-group: Indicates the auto-scaling group associated with the resource.
3. automation-tool: Specifies the automation tool used to manage the resource, such as “cloudformation”, “terraform”, etc.
4. scheduled-automation: Indicates whether the resource is part of scheduled automation tasks. Values can be “yes” or “no”.
5. continuous-integration: Indicates whether the resource is integrated with a continuous integration (CI) system.
6. automated-backup: Specifies whether automated backups are enabled for the resource.
7. auto-shutdown: Indicates whether the resource has an automated shutdown schedule.
8. auto-recovery: Specifies whether the resource is configured for automatic recovery in case of failure.
9. auto-scaling-policy: Indicates the auto-scaling policy associated with the resource.
10. automated-monitoring: Specifies whether automated monitoring is enabled for the resource.

Business Tags:

1. cost-center: Associates the resource with a specific cost center or project code for financial tracking and allocation.
2. department: Indicates the department or business unit that owns or uses the resource. Useful for chargebacks and cost allocation.
3. project: Associates the resource with a specific project or initiative within the organization.
4. business-unit: Specifies the business unit that the resource belongs to.
5. customer-id: Identifies the customer or client associated with the resource.
6. revenue-center: Associates the resource with a specific revenue center or profit center.
7. business-criticality: Indicates the business criticality of the resource, such as “high”, “medium”, or “low”.
8. contract-id: Specifies the contract ID related to the resource.
9. service-level-agreement: Indicates the service level agreement (SLA) associated with the resource.
10. business-impact: Specifies the potential business impact of the resource being unavailable.

Security Tags:

1. security-classification: Specifies the security classification or sensitivity level of the resource, such as “public”, “internal”, “confidential”.
2. compliance: Specifies the compliance status of the resource, such as “hipaa”, “pci-dss”, “gdpr”, etc.
3. backup: Indicates whether the resource is included in regular backups or not. Values can be “yes” or “no”.
4. encryption: Indicates whether the resource data is encrypted at rest and/or in transit.
5. access-control-list: Specifies the access control list (ACL) associated with the resource.
6. security-group: Indicates the security group associated with the resource.
7. firewall-rule: Specifies the firewall rule associated with the resource.
8. vulnerability: Indicates the vulnerability status of the resource, such as “vulnerable”, “patched”, etc.
9. data-classification: Specifies the data classification level of the resource, such as “sensitive”, “confidential”, etc.
10. security-policy: Indicates the security policy applied to the resource.

Tags for Classification:

1. environment: Indicates the environment the resource belongs to, such as “production”, “development”, or “staging”.
2. application: Identifies the application or project to which the resource belongs. Useful for multi-application environments.
3. region: Specifies the AWS region where the resource is located.
4. zone: Indicates the availability zone within a region where the resource is located.
5. role: Specifies the role or function of the resource within the system architecture.
6. service-type: Specifies the type of service provided by the resource, such as “web”, “database”, etc.
7. deployment-stage: Indicates the deployment stage of the resource, such as “alpha”, “beta”, “release-candidate”, etc.
8. cost-allocation: Specifies the cost allocation tag for financial tracking purposes.
9. cost-savings: Indicates whether the resource is part of a cost-saving initiative. Values can be “yes” or “no”.
10. expiration-date: Specifies the date when the resource should be reviewed or decommissioned. Useful for managing resource lifecycle.

Tagging Use Cases

Tags for AWS Console Organization and Resource Groups

• organize-resources: Use tags to group and categorize resources based on their purpose, project, environment, or any other relevant attribute. This makes it easier to navigate and manage your resources within the AWS Management Console.
• resource-group-classification: Leverage tags to create resource groups based on specific criteria, such as tagging all production resources together or grouping resources belonging to a specific department or business unit.

Tags for Cost Allocation

• cost-tracking: Assign tags to AWS resources to identify the associated costs and allocate them to specific cost centers, departments, projects, or customers. This enables you to track and analyze your AWS spending accurately.
• cost-allocation-reports: Utilize tags to filter and generate detailed cost allocation reports using AWS Cost Explorer or Cost and Usage Reports. These reports can provide insights into cost breakdowns based on different tag values.

Tags for Automation

• automated-resource-management: Use tags to trigger automation scripts, workflows, or AWS Lambda functions for managing resources automatically. For example, you can use tags to automatically start or stop EC2 instances based on predefined schedules or resource utilization.
• automated-backup-and-recovery: Tagging resources can be useful in automated backup and recovery scenarios. By assigning specific tags to resources, you can create backup policies or automation scripts that target resources with those tags for backup or recovery operations.

Tags for Operations Support

• resource-monitoring-and-alerts: Assign tags to resources to facilitate monitoring and alerting systems. Tags can be used to set up monitoring dashboards, define alarm rules, or enable automated actions based on resource state changes or performance metrics.
• incident-response-and-troubleshooting: Tags can assist in incident response and troubleshooting by providing additional context or categorization for resources. During an incident, tags can help identify affected resources, their interdependencies, or associated configurations.

Tags for Access Control

• access-control-policies: Use tags to define fine-grained access control policies through AWS Identity and Access Management (IAM). Tags can be used as condition keys in IAM policies, allowing you to grant or restrict access to resources based on their tag values.
• resource-sharing-and-cross-account-access: Leverage tags to control resource sharing and enable cross-account access. You can use tags to specify which resources are accessible to other AWS accounts or specific IAM roles based on tag-based permissions.

Tags for Security Risk Management

• security-group-and-nacl-rules: Assign tags to security groups and network ACLs (NACLs) to manage and audit security policies effectively. Tags can help categorize or identify the purpose of security rules, allowing you to track and analyze security configurations and potential risks.
• asset-inventory-and-vulnerability-management: Utilize tags to maintain an inventory of your AWS assets and support vulnerability management practices. Tags can help identify resources that require regular security assessments, vulnerability scanning, or patch management.

Creating a VPC without using the wizard

In this post we are going to create a VPC with a private and public subnet without using the wizard to get into VPC in detail and to know how it works internally.

You also can create your VPC using the wizard following the AWS tutorial Creating a VPC with Public and Private Subnets for Your Compute Environments

1. Open the AWS console and go to Services → Networking & Content Delivery section and make click on VPC.

2. On the left menu, in the VIRTUAL PRIVATE CLOUD section, click on Your VPCs option.

3. Click on the button that says Create VPC on the top side.

4. Fill the form that appears with the data below.

   Name tag: vpc-us-east-1-test-lab
   IPv4 CIDR block: 10.0.0.0/16
   IPv6 CIDR block: Amazon provided IPv6 CIDR block
   Tenancy: Default

   Note Be aware of the VPC description that appears in the form. A VPC is an isolated portion of the AWS cloud populated by AWS objects, such as Amazon EC2 instances. You must specify an IPv4 address range for your VPC. Specify the IPv4 address range as a Classless Inter-Domain Routing (CIDR) block; for example, 10.0.0.0/16. You cannot specify an IPv4 CIDR block larger than /16. You can optionally associate an IPv6 CIDR block with the VPC.

   Note By default, when you create a custom VPC, the following components are created.

   • A router
   • A main route table
   • A main ACL
   • A main Security Group

5. On the VPC section, verify that the VPC was created successfully and then make click on Subnets on the left menu.

6. Click on the button that says Create subnet on the top side.

7. Fill the form that appears with the data below.

   Name tag: subnet-us-east-1a-public-test-lab
   VPC: vpc-us-east-1-test-lab
   Availability Zone: us-east-1a
   IPv4 CIDR block: 10.0.1.0/24
   IPv6 CIDR block: Don’t Assign IPv6

8. Repeat the steps 5 and 6 in order to create a second subnet.

9. Fill the form that appears with the data below.

   Name tag: subnet-us-east-1b-private-test-lab
   VPC: vpc-us-east-1-test-lab
   Availability Zone: us-east-1b
   IPv4 CIDR block: 10.0.2.0/24
   IPv6 CIDR block: Don’t Assign

10. Go to the Subnet section and select subnet-us-east-1b-private-test-lab, then click on Actions button on the top side, then click on Modify auto-assign IP settings and finally select Auto-assign IPv4 option.

11. Go to the VPC Dashboard section and on the left menu click on Internet Gateways and then click on Create internet gateway on the top side.

12. Fill the form that appears with the data below.

    Name tag: igw-vpc-us-east-1-test-lab

13. Attach the internet gateway to the VPC vpc-us-east-1-test-lab

14. Go to the VPC Dashboard section and on the left menu click on Route Tables and then click on Create route table on the top side.

15. Fill the form that appears with the data below.

    Name tag: rtb-subnet-us-east-1a-public-test-lab
    VPC: vpc-us-east-1-test-lab

16. Repeat the steps 14 and 15 in order to create a route table for the private subnet.

    Name tag: rtb-subnet-us-east-1b-private-test-lab
    VPC: vpc-us-east-1-test-lab

17. Select the route table rtb-subnet-us-east-1a-public-test-lab and on the bottom side select routes and then click on the Edit routes button.

18. Add the following rules and then click save.

    Destination	Target
    0.0.0.0/0	igw-vpc-us-east-1-test-lab
    ::/0	igw-vpc-us-east-1-test-lab

19. Select the route table rtb-subnet-us-east-1a-public-test-lab and on the bottom side select subnet associations and then click on Edit subnet associations, select subnet-us-east-1a-public-test-lab and save.

20. Create a NAT gateway associated with the subnet subnet-us-east-1a-public-test-lab and add a tag name as ng-subnet-us-east-1a-public-test-lab

21. Select the route table rtb-subnet-us-east-1b-private-test-lab and on the bottom side select routes and then click on the Edit routes button.

22. Add the following rules and then click save.

    Destination	Target
    0.0.0.0/0	ng-subnet-us-east-1a-public-test-lab

23. Select the route table rtb-subnet-us-east-1b-private-test-lab and on the bottom side select subnet associations and then click on Edit subnet associations, select subnet-us-east-1b-private-test-lab and save.

24. Go to the VPC Dashboard section and on the left menu click on Network ACLs and then click on Create network ACL on the top side.

25. Create two ACL one for the public subnet and another for the private subnet.

    [ACL-1]
    Name tag: acl-subnet-us-east-1a-public-test-lab
    VPC: vpc-us-east-1-test-lab

    [ACL-2]
    Name tag: acl-subnet-us-east-1b-private-test-lab
    VPC: vpc-us-east-1-test-lab

26. Associate each ACL with its corresponding subnet.

27. Add to each ACL the rules you need.

28. Create two Security Groups one for the public subnet and another for the private subnet.

    [SG-1]
    Security group name: sg-subnet-us-east-1a-public-test-lab
    Description: Any description
    VPC: vpc-us-east-1-test-lab

    [SG-2]
    Security group name: sg-subnet-us-east-1b-private-test-lab
    Description: Any description
    VPC: vpc-us-east-1-test-lab

References and Resources

• Amazon Virtual Private Cloud Documentation
• Amazon Virtual Private Cloud VPC
• CIDR

AWS Naming Conventions Best Practices

A naming convention is a well-defined set of rules useful for choosing the name of an AWS resource. Ensure that your AWS resources are using appropriate naming conventions for tagging in order to manage them more efficiently and adhere to AWS resource tagging best practices.

Naming (tagging) your AWS resources consistently have several advantages such as providing additional information about the resource location and usage, promoting consistency within the selected AWS region, distinguishing fast similar resources stacks from one another, avoiding naming collisions, improving clarity in cases of potential ambiguity and enhancing aesthetic and professional appearance.

Default Pattern Components

{RegionCode}
(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)

{AvailabilityZoneCode}
([1-2]{1})([a-c]{1})

{EnvironmentCode}
(dev|test|stg|prod)

{ApplicationCode}
([a-z0-9\-]+)

{SubnetRouteCode}
(public|private)

{ResourceCode}
(ec2|alb|nlb)

VPC Naming Conventions

Default Pattern Format

vpc-{RegionCode}-{EnvironmentCode}-{ApplicationCode}

RegExp

^vpc-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-(dev|test|stg|prod)-([a-z0-9\-]+)$

Examples

vpc-us-east-1-prod-bigdataappstack
vpc-us-west-2-prod-webappstack

Subnet Naming Conventions

Default Pattern Format
subnet-{RegionCode}-{AvailabilityZoneCode}-{SubnetRouteCode}-{EnvironmentCode}-{ApplicationStackCode}

RegExp
^subnet-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-([1-2]{1})([a-c]{1})-(public|private)-(dev|test|stg|prod)-([a-z0-9\-]+)$

Examples
subnet-us-east-1-2a-public-prod-webappstack
subnet-us-west-1-2b-private-prod-databasestack

Security Group Naming Conventions

Default Pattern Format
sgr_{ResourceCode}_{RegionCode}_{EnvironmentCode}-{ApplicationCode}

RegExp
^sgr_(ec2|alb|nlb)_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
sgr_ec2_us-west-1_dev_sample-app-instance-1

EC2 Instance Naming Conventions

Default Pattern Format
ec2_{RegionCode}_{AvailabilityZoneCode}_{EnvironmentCode}_{ApplicationCode}

RegExp
^ec2_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_([1-2]{1})([a-c]{1})_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
ec2_us-east-1_2a_prod_tomcat
ec2_us-west-1_2b_prod_nodejs

PEM Key Naming Conventions

Default Pattern Format
key-RegionCode-EnvironmentCode-ApplicationCode

RegExp
^key-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-(dev|test|stg|prod)-([a-z0-9\-]+)$

Examples
key-us-east-1-prod-web-server
key-us-west-1-prod-database

ECS Cluster Naming Conventions

Default Pattern Format
ecs-cluster_{RegionCode}_{EnvironmentCode}_{ApplicationCode}

RegExp
^ecs-cluster_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
ecs-cluster_us-east-1_prod_tomcat
ecs-cluster_us-east-1_prod_micro-app

ECS Task Definition Naming Conventions

Default Pattern Format
ecstd-{RegionCode}-{EnvironmentCode}-{ApplicationCode}-{Vesion}

RegExp
^ecstd-(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)-(dev|test|stg|prod)-([a-z0-9\-]+)-v1$

Examples
ecstd-us-east-1-prod-tomcat-v1
ecstd-us-east-1-prod-micro_app-v1

ECS Service Naming Conventions

Default Pattern Format
ecs-service_{RegionCode}_{EnvironmentCode}_{ApplicationCode}

RegExp
^ecs-service_(us-east-1|us-west-1|us-west-2|eu-west-1|eu-central-1|ap-northeast-1|ap-northeast-2|ap-southeast-1|ap-southeast-2|sa-east-1)_(dev|test|stg|prod)_([a-z0-9\-]+)$

Examples
ecs-service_us-east-1_prod_tomcat
ecs-service_us-east-1_prod_micro-app

References

• Cloud Conformity VPCs
• Cloud Conformity EC2

Apache Kafka Architecture & Fundamentals Q&A

These are the answers and questions from a tech talk led by Confluent about Apache Kafka Architecture and Fundamentals.

How reliable is the data ingest to Kafka from the producer(s)?
Data can be acknowledged in the broker and also in copies (replicas) on other brokers. You have your choice of no ack, ack from the receiving broker, or ack once that one and all of the broker copies have successfully received the message.

Does Kafka require Hadoop cluster infrastructure like MapReduce, HDFS, etc.?
No, a Kafka cluster of brokers requires only Kafka, and for now, Zookeeper

Can we store in local storage or also on data lake storage?
Kafka can store on both - in memory initially and persisted on disk at an adjustable time after.

Is the retention time the same for all brokers in a cluster? Or Can we set a different retention time for each broker?
Retention time can be set per topic, not per broker.

Is there any way to keep the data rather than a generic retention time configuration setting? For example, if there is a slower moving set of topics compared to higher velocity data
Yes - retention times are uniquely configurable based on the particular Kafka architectural design.

If data expires within Broker’s local storage then does it mean it get lost? Suppose it has not been read/consumed by “all” the consumers of that data
Kafka doesn’t really support the concept of a specific piece of data having been consumed. It may be consumed by multiple consumers - or if your retention period is short and no consumers are consuming from the topic it is in, yes a particular record could be deleted without ever being consumed.

Kafka brokers needs a separated master?

I’m not sure what you mean by this question. Brokers work cooperatively in a cluster. At this time, they use Zookeeper for some coordination, although that requirement will be removed in the future.

Are there local disks SSD or hard drives?
Local disks for the broker can be of any type.

Can some other components be used instead of Zookeeper?
No but Zookeeper is set to be deprecated at some point - please see KIP 500: https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/

Can Kafka be used in on-premise setup? If so what is the minimum system requirements/how many servers are required to configure Kafka?
Kafka can be run on as few as one server, but three would commonly be ideal for failover support.

Is it kind of a layer in between source and target to stream the data directly without doing any data transformation?
I’m a little unclear on the intent of this question. Yes, data can pass through Kafka without any transformation. Because the producers of the data and the consumers of the data are decoupled, and data is persisted on the cluster, data can be consumed by multiple different consumers over time. So data could be transported to many targets.

If a message is successfully consumed, it is stored till the retention time any way?
Yes! Once received, messages are immutable until the retention period expires.

Does the data stay on local storage for 5 days ?
The retention time, the time that data is preserved on local storage, is configurable by topic - so data could stay on one topic for 5 days and another for a much longer or shorter time. It is not uncommon to have transient data - that is just being passed through Kafka, to be retained for only a few hours. Some businesses store data on some topics for much longer - weeks, months, years or forever - depending on their business needs.

Is there a limit to the number of topics that a Broker node can handle?
There are best practices - but this is more of a logical limit, i.e., how many topics to keep track of. There needs to be enough, but not too many depending on the applications using Kafka.

Is Zookeeper platformed within a Kafka instance or do I have to stand up and operate a separate zookeeper instance to keep my cluster operational?
Yes, Zookeeper needs to be stood up when installing Kafka currently - but as Joe said it will be going away per KIP 500. Please see: https://www.confluent.io/blog/removing-zookeeper-dependency-in-kafka/

Can we have authorization enabled for consumer connecting using rest proxy having SASL_SSL security with SCRAM mechanism.?
This mechanism is for authentication - for authorization, you need to use ACLs (Access Control Lists) or the new RBAC (Role Based Access Control) which is a feature of the Confluent Platform, not open source Kafka.

Can kafka be used as permanent data store?
Yes! Kafka can be used as a permanent store, ksqlDB allows it to even be used as a form of a database!

Can we use a 3rd party enterprise metadata instead of the zookeeper?
At this time, Zookeeper is required. In the near future, Kafka clusters will be self managing and the Zookeeper requirement will be removed - but will not need any other 3rd party tools.

What does adding consumers mean? Does adding consumer mean adding a new node to the cluster?
Adding consumers means adding more applications that are reading from the cluster (brokers).

When kafka store data on local disk , what is preferred 1) one disk with 15tb of size or 2) 15 disks with 1tb size each?
OF course, that depends. Kafka is a distributed event system, so you can have multiple Kafka Brokers and then each will need to have its own local disk or disks.

Is it common to have a consumer that is also a producer? For example, a process that takes incoming messages and “enriches” them for later consumption by a different process?
Yes, it is common - in fact, any stream processing application both consumes data from Kafka and then producers transformed data back to Kafka.

If we request to produce or consume.. will it go to zookeeper or broker?
All clients - producers and consumers - communicate only with Kafka Brokers.

Can a message be consumed more than once by the same consumer?
Generally an individual consumer will only consume any message one time - however, a consumer can set its consumer offset - the point at which it is reading from a topic - and could then re-read any number of messages.

How is KAFKA different from a Message Queue MQ Series or ESB solutions?
The main difference is independence between the producer of messages and consumers of messages. The broker is in the middle as an independent handler of either 1) producer write or 2) consumer read requests in a given transaction.

What are the data formats that are supported by Kafka? Also where does the transformations happen for business rules?
Kafka supports many data formats - there are many serializers of data types to include string, long, int, etc. Also there are many Schema formats to include Avro, Protobuf, and JSON Schema.

Any chance of message will be lost in Kafka?
It is possible for a message to be lost when produced to Kafka - you should look at the producer ACK configuration settings. If a producer sets ACKS=0, then it will just send a message and not care if it is received by a Broker or not.

I understand that the storage is always held by Zookeeper. Is it true or Can I set for instance Oracle as the persistence?
Message data in a Kafka cluster is only persisted on local Kafka Broker storage. It is not held by Zookeeper nor can you persist it to Oracle. You could, of course use Connect to move data from Kafka storage to Oracle if you wish.

Any scenarios where we don’t need Zookeeper or we should always use it? We have an environment implemented by consultants that does not use Zookeeper. There are also some articles on the Internet that talks about not using Zookeeper.
With the current release of Kafka, you must use Zookeeper. That requirement will be removed in a future release as Kafka clusters move to self management.

If a broker dies, is the producer data on broker lost, or is it replicated on other brokers as well so it is never lost?
When you create a topic, you define the amount of replication that topic requires. It is typical to have 3 replicas for a topic - which means that even if 2 brokers crash you can continue to produce to and consume from your topic without data loss.

Is retention time the only way to have data removed/dropped from the cluster or can it be set in a way that if a consumer has consumed the data then it can be dropped from the cluster?
Kafka doesn’t keep track of whether data has been consumed or not - there can be multiple consumers doing different tasks with the same data, or one consumer may choose to re-consume an entire topic - so retention time controls how long data will be stored on Kafka - on a per topic basis.

Can I add in Kafka code to enrich new data with some topic? Or need do outside in another process?
Yes! This is perfect for a Streams application if more than just a minimal addition is being done.

Would it be possible to stream the messages of size 25MB via kafka? Any limitations?
The default maximum message size on Kafka is 1 MB. While you can configure Kafka to use larger messages, it is not optimal to do so.

Can two producers write to the same Topic? Or is Topic specific to a Producer?
Yes - that is part of the beauty of Kafka! Multiple producers. And the same with consumers too! :-)

Is the kafka architecture based generally enough to handle multiple use cases, or is the architecture very bespoke depending on each customer use case?
Excellent question! For basic applications, it is general enough - but for more complicated ones, that for example may need to scale, it would need to become bespoke. Rather, the larger the Kafka architecture the more bespoke it generally needs to be.

What data producers will not connect to kafka?
To produce to Kafka, your producer needs to conform to Kafka protocols. The native APIs are Java based, but there are libraries for many other languages. If you are not able to use any of the available languages, you can send data to Kafka through the REST Proxy, which can communicate with any process that can send an HTTP REST request. You can also use Kafka Connect to move data from many standard data sources into Kafka.

Does data also split the data in between multiple partitions?
A Kafka Producer will be able to send data to any partition of a topic - from the metadata it receives from a Kafka Broker, it knows where the leader broker is for each partition of a topic and it uses a partitioner to determine which partition to send data to.

How to decide the partition size for a topic ? What happens if we resize the topic once it is created with “N” partitions?
Great question that has a lot behind it. There are some White Papers and/or blog posts on the Confluent and Kafka websites that help address this. Much of what Joe is explaining is the basis for that! Yes, it’s possible to resize as an operation, but with some caveats.

If I need to post transactions extracted from 500 tables into kafka, 1.is that a good idea to create 500 topics for each table, to make sure the consumers reads it in the same insertion order
Good point! Order is something that requires overhead to enforce in Kafka. So yes, you may end up needing to create individual topics to ensure order. But for what you reference, please check out Kafka Connect which may be able help handle that better.

Presenter mentioned that partitions are striped across brokers, but I thought broker cluster used leader and standby replicas? Are the brokers not replicated, but working in concert?
Each topic may be broken into a number of partitions - you define the number when you create a topic. Those partitions can reside on different brokers - and each broker will receive data appropriate for its own partitions. Partitions can also be replicated - so each partition will have a leader broker and some number of follower brokers.

Are there any advantages of using just one single topic, collecting all those 500 table messages? I am afraid this would mix up the insertion order when consumers read 500 tables data from one single topic.
Yes, to ensure order for sure you can use single topics each having single partitions. But there may be a way to design your application, depending on its needs, that could be more efficient - please check out the white papers and blog entries on confluent.io!

Compaction retention policy does or does not allow for mutation of messages?
Messages are not mutated once they arrive at a Kafka broker - that is an inherent part of the design - all messages are immutable. However, messages can be removed from a topic if it is compacted - but the content of the message will not change.

Can I change the sequences of the messages in the logs?
No - and yes. No you can’t change what’s been written to a log as it’s immutable. But you can further process what’s been written into a log, where you read it - and maybe even process it or add to it from say a DB - then write it to a different log. Pretty cool?! This is known as a Streams application.

So, the data written by producers need to be schema less right ? And consumer needs to understand that format right?
The data needs to be serialized to Kafka on the way in (produce), and deserialized on the way out (consume). You can use schemas to help control this.

How many offsets we can have in a stream?
The offset number for a message in a partition is a 64 bit number, so you can have 2 to the 64th power number of messages in a partition. You can define your topic to have many partitions. A stream can have as many messages as the underlying topic.

Can I run Kafka on raspberry pi with small memory foot print. What is it’s memory foot print. I assuming memory foot print is configurable .please confirm.
Yes! Kafka will run on Raspberry Pi. Sorry I don’t have the memory requirements, you should be able to find them at https://kafka.apache.org/ and please also check out http://confluent.io !

Is there a chance of messages being out of sync, means the timestamp is not in order?
As a producer you can define what the timestamp is - you could use the event time, when the event actually took place, or the time the message is produced by the producer or the ingest time, when the message is written by the Broker. So, yes, you could have messages in a topic that have timestamps that are out of order.

Is a log file equivalent to a segment? Partition has 1 to n log files
Raphael, yes! log file = segment.

How to avoid multiple deliveries of same messages?
Simply you can turn on idempotence in your Kafka client code. You can also use Exactly Once Semantics - also known as EOS: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Can we use Kafka to process a large cuantiti of daily images, pdf documents as messages and storage?
Because the default message size is 1 MD in Kafka, it is not unusual to have image or pdf data directly in the messages. That said, you could chunk large messages - but then you are responsible for correctly reassembling them. Or, you can pass references in a message to images or pdf documents that are stored externally.

How do you typically handle duplicate messages? Let’s say we need to do some operation say write to a DB, but a consumer can receive the same message twice..And we aren’t able to know whether the item is already written to the DB or not when we receive the duplicate msg.
Please check this out - explains a lot of what I think you may want to know: https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/

Are there any limitations if we use Python ? Does streaming support Python?
Python is a supported language for writing Kafka producer and consumer code. However Java is the closest to Kafka’s source code.

How to create an AWS billing alarm

1. Set the billing preference, check and enable the option to receive billing alerts.
2. Go to Cloud Watch console.
3. Change the region, make sure your AWS region is settled to US EAST (N. Virginia).
4. Create an alarm, click on Alarms on the left side of the panel and in the dashboard that pops up click on the orange button that says “Create alarm”.
5. Select a metric, in the panel that pops up, click the button that says “Select metric”. This will pop up a window that allows you to choose which type of service you would like to use. Select Billing
6. Select the service, in the next window you can choose the total estimated charged on your account.
7. Select the period of time that the alarm will be activated.
8. Set the threshold.
9. Create a notification using SNS, you just click on Create new topic give it a name and enter the email where you want to receive the notification.
10. Add a description.
11. Click on Preview and create

Simple Storage Service (S3)

Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance. This means customers of all sizes and industries can use it to store and protect any amount of data for a range of use cases, such as websites, mobile applications, backup and restore, archive, enterprise applications, IoT devices, and big data analytics.

Amazon S3 provides easy-to-use management features so you can organize your data and configure finely-tuned access controls to meet your specific business, organizational, and compliance requirements.

Amazon S3 is designed for 99.999999999% (11 9’s) of durability, and stores data for millions of applications for companies all around the world.

You can use S3 to store and retrieve any amount of data at any time, from anywhere on the web.

The basics of S3 are

• S3 is object-based and allow you to upload files.
• Files can be from 0 Bytes to 5TB
• There is unlimited storage
• Files are store in buckets. Think about bucket as folders.
• S3 is a universal namespace, this means that the name of the bucket must be unique globally.
• When you upload a file to S3, you will receive a http 200 code if the upload was successful

S3 Objects

Think of objects just as files. Objects consist of the following:

• Key: This is simply the name of the object.
• Value: This is simply the data and is made up of a sequence of Bytes.
• VersionId: Used for versioning
• Metadata: Data about the data you are storing
• Sub-resources: ACLs, torrent

S3 data consistency

• Read after write consistency for PUTS of new objects, this means that if you upload a file for first time you will be able to read it immediately.

• Eventual consistency for overwrite PUTS and DELETES, this means that if you upload or update or delete an existing file and try to read it immediately you may get the older version or you may not, basically changes to objects can take a little bit of time to propagate.

S3 features

• Tired storage availability (S3 standard, S3-IA, S3-OneZone-IA, S3 Intelligent Tiering, S3 Glacier, S3 Glacier deep archive).
• Lifecycle management.
• Versioning.
• Encryption.
• MFA for delete.
• Secure your data using access control lists ACLs and bucket policies.

S3 storage classes and tiers.

• S3 Standard: 99.99% availability and 99.99999999999% durability, store your objects redundantly across multiple devices in multiple facilities and is designed to sustain the loss of two facilities concurrently.

• S3-IA Infrequent Access: Is for data that is less frequently accessed but requires rapid access when needed. It has a lower fee than S3 Standard, but you will be charged a retrieval fee.

• S3 One Zone Infrequent Access: For where you want a low-cost option for infrequently accessed data, but that does not require the multiple availability zone data resilience.

• S3 Intelligent-Tiering: Designed to optimize costs by automatically moving data to the most cost-effective access tier, without performance impact or operational overhead.

• S3 Glacier: Is a secure, durable and low-cost storage class for data archiving. You can reliably store any amount of data at costs that are competitive with or cheaper than on-premise solutions. Retrieval times is configurable from minutes to hours.

• S3 Glacier Deep Archiving: Is the amazon S3 lower-cost storage class where a retrieval time of 12 hours is acceptable.

S3 performance

Your application can achieve at least 3,500 PUT/COPY/POST/DELETE or 5,500 GET/HEAD requests per second per partitioned prefix. There are no limits to the number of prefixes in a bucket. You can increase your read or write performance by using parallelization. For example, if you create 10 prefixes in an Amazon S3 bucket to parallelize reads, you could scale your read performance to 55,000 read requests per second. Similarly, you can scale write operations by writing to multiple prefixes.

Optimizing Amazon S3 performance

S3 Security and Encryption

By default, all newly created bucket are private, you can setup access control to your bucket using:

• Bucket policies at bucket level.
• Access Control List ACLs at objects level.

S3 buckets can be configured to create access logs which log all requests made to the S3 bucket. This can be sent to another bucket and even another bucket in another account.

Encryption in transit

Encryption in transit is achieved by SSL/TLS

Encryption at rest (Server Side)

• S3 Manage Keys SS3-S3: Where Amazon manages all the keys
• AWS Key Management Service SSE-KMS: Where you and Amazon manage the keys
• Server Side Encryption Where you provide the KEY-SSE-C and give to Amazon your own keys that you manage.

Encryption at rest (Client Side)

Where you upload your files already encrypted using your own keys.

S3 Versioning

• Stores all versions of an object (Including all writes and even if you delete an object)
• Once enabled, versioning cannot be disabled but only suspended.
• Versioning is integrated with lifecycle rules.
• Versioning has MFA capabilities, which uses multi-factor authentication to provide an additional layer of security.

S3 Lifecycle management

Uses lifecycle rules to manage your objects, which define how Amazon S3 manages objects during lifetime. Lifecycle rules enable you to automatically transition objects among the different storage classes and tier from S3 to Glacier. Using a lifecycle rule you can automatically expire objects based on your retention needs or clean up incomplete multi-parts uploads.

Sharing S3 buckets and objects across accounts.

There three ways to share buckes across accounts.

• Using bucket policies and IAM Applies across the entire bucket and provide programmatic access only.
• Using ACLs and IAM Applies to individual objects and provide programmatic access only.
• Cross-Account IAM Roles Provides programmatic and console access.
• See Example walkthroughs: Managing access to your amazon S3 resources

S3 Cross region replication

Replication enables automatic, asynchronous copying of objects across Amazon S3 buckets. Buckets that are configured for object replication can be owned by the same AWS account or by different accounts. You can copy objects between different AWS Regions or within the same Region.

• Versioning must be enabled on both the source and the destination buckets.
• Files in an existing bucket are not replicated automatically.
• All subsequent updated files will be replicated automatically.
• Delete markers are not replicated.
• Deleting individual versions or delete markers will not be replicated.

To enable object replication, you add a replication configuration to your source bucket. The minimum configuration must provide the following:

• The destination bucket where you want Amazon S3 to replicate objects
• An AWS Identity and Access Management (IAM) role that Amazon S3 can assume to replicate objects on your behalf

S3 Transfer acceleration.

S3 transfer acceleration uses the Cloud Front edge network to accelerate your uploads to S3 instead of uploading directly to your S3 bucket, you can use a distinct URL to upload your files to an edge location which then transfer that files to S3 using the Amazon enhanced network.

Enables fast, easy and secure transfer of files over long distances between your clients and an S3 bucket. S3 transfer acceleration takes advantage of Amazon CloudFront globally distributed edge locations as the data arrives at an edge location and then is routed to an S3 bucket over an optimized network path.

You will get a distinct URL to upload your files, example: buckt-name.s3-accelerate.amazonaws.com

How is S3 billed

• For storage.
• For requests.
• For storage management.
• For data transfer.
• For transfer acceleration.
• For cross-region replication.

AWS Identity and Access Management (IAM)

Is a web service that helps you securely control access to AWS resources. You use IAM to control who is authenticated (signed in) and authorized (has permissions) to use resources. IAM allows you to manage users and their level of access to the aws console.

IAM Key terminology

• Users End user such as people, employees of an organization, etc.
• Groups A collection of users. Each user in the group will inherit the permissions of the group.
• Policies Policies are made of documents, called policy documents. These documents are in JSON format and they give permissions as to what a user/group is able to do.
• Roles You create roles a then assign them to AWS resources.

IAM Tips

• IAM is universal. It does not apply to regions at this time.
• The root account is simply the account created when first setup your AWS account. It has complete admin access.
• New users has NO permissions when first created.
• New users are assigned and Access Key and an Secret Key when first created.
• You cannot use the access key and the secret key to login into the console.
• You can view the access key and the secret key once, if you lose it you have to regenerate them.
• Always set up Multifactor Authentication especially for the root account.
• You can create and customise your own password rotation policies.

IAM Best practices

• Lock Away Your AWS Account Root User Access Keys You use an access key (an access key ID and secret access key) to make programmatic requests to AWS. However, do not use your AWS account root user access key. The access key for your AWS account root user gives full access to all your resources for all AWS services, including your billing information. You cannot reduce the permissions associated with your AWS account root user access key.

• Create Individual IAM Users Don’t use your AWS account root user credentials to access AWS, and don’t give your credentials to anyone else. Instead, create individual users for anyone who needs access to your AWS account. Create an IAM user for yourself as well, give that user administrative permissions, and use that IAM user for all your work. For information about how to do this, see Creating Your First IAM Admin User and Group.

• Use Groups to Assign Permissions to IAM Users Instead of defining permissions for individual IAM users, it’s usually more convenient to create groups that relate to job functions (administrators, developers, accounting, etc.). Next, define the relevant permissions for each group. Finally, assign IAM users to those groups.

• Grant Least Privilege When you create IAM policies, follow the standard security advice of granting least privilege, or granting only the permissions required to perform a task. Determine what users (and roles) need to do and then craft policies that allow them to perform only those tasks.

• Get Started Using Permissions with AWS Managed Policies To get started quickly, you can use AWS managed policies to give your employees the permissions they need to get started. These policies are already available in your account and are maintained and updated by AWS. For more information about AWS managed policies, see AWS Managed Policies.

• Use Customer Managed Policies Instead of Inline Policies For custom policies, we recommend that you use managed policies instead of inline policies. A key advantage of using these policies is that you can view all of your managed policies in one place in the console. You can also view this information with a single AWS CLI or AWS API operation. Inline policies are policies that exist only on an IAM identity (user, group, or role). Managed policies are separate IAM resources that you can attach to multiple identities. For more information, see Managed Policies and Inline Policies.

• Use Access Levels to Review IAM Permissions When you review a policy, you can view the policy summary that includes a summary of the access level for each service within that policy. AWS categorizes each service action into one of five access levels based on what each action does: List, Read, Write, Permissions management, or Tagging. You can use these access levels to determine which actions to include in your policies.

• Configure a Strong Password Policy for Your Users If you allow users to change their own passwords, require that they create strong passwords and that they rotate their passwords periodically. On the Account Settings page of the IAM console, you can create a password policy for your account. You can use the password policy to define password requirements, such as minimum length, whether it requires non-alphabetic characters, how frequently it must be rotated, and so on. For more information, see Setting an Account Password Policy for IAM Users.

• Enable MFA For extra security, we recommend that you require multi-factor authentication (MFA) for all users in your account.

• Use Roles for Applications That Run on Amazon EC2 Instances Applications that run on an Amazon EC2 instance need credentials in order to access other AWS services. To provide credentials to the application in a secure way, use IAM roles. A role is an entity that has its own set of permissions, but that isn’t a user or group.

• Use Roles to Delegate Permissions Don’t share security credentials between accounts to allow users from another AWS account to access resources in your AWS account. Instead, use IAM roles. You can define a role that specifies what permissions the IAM users in the other account are allowed. You can also designate which AWS accounts have the IAM users that are allowed to assume the role.

• Do Not Share Access Keys Access keys provide programmatic access to AWS. Do not embed access keys within unencrypted code or share these security credentials between users in your AWS account. For applications that need access to AWS, configure the program to retrieve temporary security credentials using an IAM role. To allow your users individual programmatic access, create an IAM user with personal access keys.

• Rotate Credentials Regularly Change your own passwords and access keys regularly, and make sure that all IAM users in your account do as well.

• Remove Unnecessary Credentials Remove IAM user credentials (passwords and access keys) that are not needed.

• Use Policy Conditions for Extra Security To the extent that it’s practical, define the conditions under which your IAM policies allow access to a resource. For example, you can write conditions to specify a range of allowable IP addresses that a request must come from. You can also specify that a request is allowed only within a specified date range or time range. You can also set conditions that require the use of SSL or MFA (multi-factor authentication). For example, you can require that a user has authenticated with an MFA device in order to be allowed to terminate an Amazon EC2 instance. For more information, see IAM JSON Policy Elements: Condition in the IAM Policy Elements Reference.

• Monitor Activity in Your AWS Account You can use logging features in AWS to determine the actions users have taken in your account and the resources that were used. The log files show the time and date of actions, the source IP for an action, which actions failed due to inadequate permissions, and more.

• Video Presentation About IAM Best Practices

References

• AWS Identity And Access Management