Tuesday, January 23, 2024

How to read

When I was first starting to read research papers in mathematics, I got some great advice from one of my professors.  He said, "Always have paper and pencil with you when you read a paper.  Read a little bit and then try to write the next part yourself.  Look at what is written in the paper as a sequence of hints.  That's all you are going to get.  You need to fill in the details yourself and if you can't do that, you have not understood the paper."   Over the years, I have realized that while research mathematics is kind of an extreme case, the same actually applies to any challenging text. So here is "the method":

  1. Read a sentence or paragraph or however much you need to get an idea.
  2. Write or say to yourself what you think is going to come next.
  3. Start from the beginning and read through the next chunk.  Compare your continuation to what actually came next. 
  4. Go to 2
You end up re-reading the whole piece many times this way.  For long things, use major breaks like chapters or whatever to limit the look back.

If you are trying to really learn the material, you can do the whole process repeatedly.  In that case, the checking in step 3 should start to show less and less divergence, mostly just style or sequencing.  You need to be careful though not to devolve into memorization.  You want to actually come up with the ideas that come next, not the words.

I do this kind of thing when I read hard material of any kind - not rigidly and sometimes changing chunks around.  If I go slowly enough, unless the material is really over my head or I am lacking needed background or something, I always end up feeling like I have had the ideas that the author was trying to convey.  That usually means that I can start to apply the ideas myself.

Sunday, January 14, 2024

Why I love mathematics

Millie and Al's https://www.flickr.com/photos/27480193@N05/

I love mathematics because it never says one thing and does something else. If it ever seems to do that, it is always because I am missing some idea. I never stay mad at mathematics.

I love mathematics because it is always there, waiting for me. It will always be there even if I don't jump on it right now. It worn't run away or turn into some not fun thing. When I go back over mathematics that I haven't looked at in a while, it's like going back to the old neighborhood and having that warm and happy feeling you get when nothing has changed. 

I love mathematics because it loves me. Mathematics has infinite patience for me. I can be arbitrarily stupid for arbitrarily long. Mathematics keeps the light on for me.

I love mathematics because it surprises me and makes me think differently all the time. I feel like Aeneas in the world of mathematics, constantly meeting monstra mirable dictu, but without the carnage.

Friday, September 15, 2023

How not to get breached (too badly)

The other day, someone asked me if I had ever experienced a major security breach. They were shocked when I responded that I had not. That is because I have been a CTO for the past 18 years, including stints at some bleeding edge SaaS companies that were constantly under attack. I have managed many security incidents, but none that resulted in a major breach. While of course it is possible and even likely that dumb luck has played a role in my success, I think that the following things have certainly helped. Each of the imperatives below have been important to me. I have never achieved perfection in any of them, but I have never stopped pushing.  The imperatives are written from the standpoint of a CTO; but anyone who cares about security can use them to help protect their company.

1. Know your risks and be honest and transparent about them

You should always have a top n list of key risks that you are worried about, and n should be less than or equal to 10.  You should have a weekly conversation with your Security officer that reviews each risk, mitigation plans, compensating controls and any help that the security team needs managing it.  It is critical that these conversations be open, honest, comprehensive and engaged. If you can't understand some of the technical security content, you need to study it and keep asking questions until you understand it.  You need to build a culture on your team that does not sugar coat or hide risks and your response to learning about them needs to be consistently positive, supportive and action-oriented.  You need to never appear to be annoyed by risks or angry at those who report them.  You also need to review critical risks at least quarterly with your partners on the executive team.  These reviews need to be open, interactive, transparent and engaging.  Your objective is not to show that you have things under control, to justify investment or to cover your ass.  Your objective is to proactively cover their ass.  If you do a good job clearly representing risks, mitigation plans and compensating controls, you will get funding and leadership support as a side effect. 

2. Focus on actual risk

Don't let compliance, vendors, talking heads or your own cool ideas distract you from systematically reducing risk.  If you do a good job identifying and managing risk, you will achieve compliance as a side effect.  Attackers don't care how "buttoned up" your security program looks or how beautifully illustrated your risk registry is.  What matters is where you are weak and what you have done to compensate for your weaknesses.  Note that this is exactly the same thing that auditors care about.  Try to put every possible cycle into things that significantly reduce actual risk.  Compensating controls can be ugly and low tech, but they can save your ass.  I am certain that some of the ugliest and most cumbersome shims that I have put in while working on permanent solutions have saved me and my companies from great harm.  

3. Pull on every thread

When you have evidence of an attack or vulnerability, make sure that you investigate everything fully. Don't just celebrate having thwarted or annoyed the attackers into leaving. You need to have people whose job it is to investigate security incidents and you have to give them the time, tools and access to do their jobs.  Ask probing questions and don't assume that your initial analysis of an anomaly is correct.  I wrote a few years back about fully solving problems.  That same thinking applies here - especially the part about "widening bugs."

4. Make security@ a welcoming and helpful place

Respond kindly and helpfully to all security-related questions or concerns from employees.  Don't put the burden on the reporter to establish that an issue is worth investigating.  You want them to come back, not to feel stupid or ignore things.  You also want them to learn.  The most important single thing that any company can do to reduce security risk is to develop or acquire high-quality security awareness training and make sure that everyone takes it.  If your company produces any kind of software (including for internal use), make sure to also acquire - and require - high-quality secure application development training.  It's very important that all training, consulting, reviews and standards promoted by your security team be of the highest quality and backed by friendly people who understand your business and are willing to patiently answer questions and help people understand security concepts.

5. Stay current

Attack types and vectors change all of the time. Many of these changes have no impact on your environment, but you need to make sure you see and assess impact of new threats as they emerge. Here again, you need to have people whose job includes monitoring security advisories and assessing their impact on you.  You need to make sure that you understand clearly what the advisories and your team are telling you.  That means you need to devote a significant amount of your professional development time to keeping up with the changing threat landscape.  As a CTO, you have a unique vantage point that virtually nobody else in the company has.  It is critical that as new threats emerge, you personally think through their implications across your technology estate.  Just as you and your team need to stay current on the threat landscape, so your systems need to remain current in terms of patch levels.  If you do not have fully automated, short-cycle patch deployment capability, this needs to be put in place.  Even if it is in place, it needs to be actively maintained and continuously exercised and you need to eliminate the things that can't be patched.

6. Do real diligence on suppliers

Some companies have well-established supplier risk management capabilities.  Even in those companies, however, sometimes the level of technical security diligence is not where it needs to be.  You need to step back from the questionnaires and boilerplate RFP responses and think clearly about where the material risks are with suppliers and dive deeply into what controls you and they have in place to mitigate them. Again, your vantage point as CTO is critical here.  You can't count on somebody else's checklist to see the full picture.  You need to direct your best spidey-sense toward where vendors may be weak and adaptively probe to make sure that you have discovered all of the risks.  Be especially careful with low-cost vendors.  Finally, make sure that you regularly review vendors' security posture.  If a vendor is acquired or the product or service you are using is slated for end of life, that should trigger a special review and contingency plan.

7. Use architecture as a weapon

Everyone knows that it's way less effective to try to add security to a naively implemented system.  On the opposite side of this are systems that deliver security "natively," exposing services and features in a way that is hostile to attackers.  Zero trust and end-to-end encryption are examples of this.  They are baked into the architecture and inherently hostile to attackers.  Really effective, dynamic and fast credential management and revocation is another great weapon.  The same service that allows applications to quickly get keys and secure communications can help you quickly respond to an attack or vulnerability.  Constantly push for the architecture win-wins that make your systems both more robust and more secure.

8. Constantly question privilege

Make least privilege an organizing principle.  Start with your own.  Do you really need production access?  You are a fat target.  Don't have anything valuable on your laptop and don't let your account be a valuable attack vector.   The same applies to everyone in your company and every process running in your environment.  The less they can do, the less valuable they are as attack vectors.  Constantly ask why individuals or processes need the access that they have and constantly prune.  Just like patching, if your environment does not have the automation, test environments or other infrastructure required to stop individuals or batch jobs from having to have weakly limited production access, you need to fix that.  If offboarding is not bulletproof when it comes to system access, make it so.  Now.  This sometimes looks daunting or even impossible to fix.  Trust me, it never is.

9. Don't acquire, decrypt, transmit or store data that you don't need

I often make the joke that the most secure and highly available system is the null system - the system that does nothing.  To deliver business value, systems need to access and process data.  But many can fully achieve their objectives with a much lighter touch on data.  Here I am reminded of an army adage about conserving energy quoted by Colin Powell in his awesome leadership book:  "Don’t run if you can walk; don’t stand up if you can sit down; don’t sit down if you can lie down; and don’t stay awake if you can go to sleep."  Here is my version for data processing:  Don't receive if you can do without; don't decrypt if you don't need to see;  don't store if you don't need to persist;  don't transmit if can omit.  The wonderful thing about modern distributed architectures is that they don't force you to create massive centralized honey pots of raw data.  So don't.  

10. Don't assume that any zone, subnet, vm or other subsystem can be hardened

I saved the best one for last.  It never ceases to amaze me how coming on 40 years now after the famous Gage/McNeally pronouncement that "the network is the computer," people keep thinking that they can somehow wall off little islands of safety and security.  You need to disabuse yourself and your team of that archaic fantasy.  You need to constantly assume that the bad guys are inside "your" network and focus on limiting what they can do once inside.  Like multi-factor authentication, strong crypto keys and end-to-end encryption, network controls are good architectural weapons, but they are just one weapon.  Like the others, they make your estate a more hostile place for attackers and slow them down; but they need the others behind them.

Some of these imperatives might seem to limit or constrain what you can do with your products or how fast you can move.  They absolutely don't have to.  For example, number 9 does not say you can't store any data. It just says you should limit what you store.  And guess what, if you do the hard thinking in system design to limit things to what you actually need, you can move faster and do more.  Once you get hard core about 8, you will also pick up speed and value because the only way to do it is to automate.  The win-wins in 7 really are all over the place and once you get that thinking baked in, first in your own mind and then in your team and entire company, you will get benefits way beyond just being a harder target.

Saturday, November 28, 2020

Governance and Humility

I recently listened to Sen. Ben Sasse read his own challenging book, Them.   In a discussion of the origins of American democracy, he mentions the following quote from James Madison:
If men were angels, no government would be necessary. If angels were to govern men, neither external nor internal controls on government would be necessary. In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself.
The core idea that the human fallibilities that make governance necessary can exist in the governing as well as the governed applies to a lot of other settings too.

In corporate settings, corporate boards provide oversight to executives who lead teams.  Great teams require little directive governance.  Great leaders provide vision and perspective, clear away obstacles and make sure that the right conversations happen at the right times with the right people involved.  Key conversations involve them personally and their own ideas and behaviors are improved as a result.  Sometimes directive leadership is needed or decisions have to be made that can't practically involve the team, but when that happens, great leaders listen carefully and actively solicit opinions of others.  And when they do stupid things, healthy organizations gently but firmly move them back onto a better path.

The basic idea that we are better off when we allow and encourage others to help us overcome our individual fallibilities is common to all of these settings and is at the core of Madison's vision, which Sasse is rightly challenging us to re-embrace.

To do that, we have to restore the humility and consideration for others that in our better moments has characterized American culture and political dialogue.  Sasse illustrates this with another powerful quote, this one from Judge Learned Hand, given in a speech called "The Spirit of Liberty" in 1944:
What do we mean when we say that first of all we seek liberty? I often wonder whether we do not rest our hopes too much upon constitutions, upon laws and upon courts. These are false hopes; believe me, these are false hopes. Liberty lies in the hearts of men and women; when it dies there, no constitution, no law, no court can save it; no constitution, no law, no court can even do much to help it...The spirit of liberty is the spirit which is not too sure that it is right; the spirit of liberty is the spirit which seeks to understand the minds of other men and women; the spirit of liberty is the spirit which weighs their interests alongside its own without bias; the spirit of liberty remembers that not even a sparrow falls to earth unheeded; the spirit of liberty is the spirit of Him who, near two thousand years ago, taught mankind that lesson it has never learned, but has never quite forgotten; that there may be a kingdom where the least shall be heard and considered side by side with the greatest.
Hand and Sasse both ground their commitment to humility and service in their religious views, but the idea of liberty, which both see as core to the idea of America, includes a deep commitment to religious freedom and tolerance for religious views different from their own.

Both also understand that governance cannot bring about the change in perspective needed to right sick organizations or societies.  That change has to happen in the hearts and minds of people.  Political leaders like Sasse can make a difference by setting good examples and I applaud him for writing the book and regularly speaking on what we need to do as a nation to heal; but to "make America great again" we have to all individually become humble again and start really listening to one another.

And that greatness will not be the greatness of a nation with borders and parochial interests, but the greatness of an idea - the idea that all human beings are created equal with inalienable rights.  That idea extends beyond any national borders and whatever greatness America ever had lies in our commitment to that idea.

Sunday, October 11, 2020

A little dusting vs. getting buried - why variance matters in ranking risks

Some simple risk management frameworks recommend stack ranking risks based on expected loss should they materialize.  The simple formula $L(r) = P(r) E(C(r))$ is sometimes used, where $P(r)$ is the probability that a risk event of type $r$ materializes and $E(C(r))$ is an estimate of the cost associated with such an event.  So for example, if the cost of a data breach at a company is estimated to be $\$1m$ and the estimated probability of this happening is $.01$, then $L$ is in this case estimated to be $\$10k$.  Given estimates of $P(r)$ and $E(C(r))$ for the different risks faced by a company or individual, a natural way to rank them is by $L$ values.  The problem with this approach is that it fails to take into account the variance in $C$ and it also puts too much emphasis on the point estimate of $L$.

Consider the following example.  Suppose that a company has one risk $r$ with $P(r) = 0.01$ and $E(C(r)) = \$1m$ and another risk $s$ with $P(s) = 0.01$ and $E(C(s))= \$800k$.  Suppose further that a realized loss of more than $\$1.2m$ is catastrophic for the company, meaning losses of this amount or greater cannot be absorbed.  If $C$ has no variance for both $r$ and $s$, neither is a catastrophic threat and it makes sense to prioritize mitigating $r$ over $s$.

The problem is that in real world situations, $C$ always has variance and when what you really care about is guarding against large losses, that variance changes the equation.  In the example above, suppose that $C(r)$ is normally distributed with mean $1m$ and standard deviation $100k$ but $C(s)$ has a uniform distribution with minimum $0$ and maximum $1.6m$.  This means that the values of $C(r)$ should follow a bell-shaped curve clustering around the mean $E(C(r)) = 1m$ but the values of $C(s)$ are randomly spread across the interval $[0,1.6m]$ with no range of values any more likely than any other.

The probability that $r$ results in a catastrophic loss under these assumptions is $P(r) P(C(r) > 1.2m) = .01 \times P(N(1, 0.1) > 1.2)$ where $N(\cdot,\cdot)$ is the normal distribution.  Now $P(N(1, 0.1) > 1.2)$ is approximately $0.023$, so that means the probability that $r$ unmitigated will result in a catastrophic loss is approximately $0.01 \times 0.023 = 0.00023$.

For $s$, $P(C(s) > 1.2) = (1.6 - 1.2) / 1.6 = 0.25$, so the probability that $s$ results in a catastrophic loss is $0.01 \times 0.25 = 0.0025$, which is more than ten times higher. So if what the company wants to do is to minimize the probability of catastrophic loss, $s$ is actually a much more important risk to mitigate.  

The practical problem is that in general the distributions of the cost functions are unknown as are their expected values.  But just asking the question of how bad a loss can be and how likely a tail event is can lead to better risk prioritization decisions.  The example above is mathematically extreme.  The uniform distribution is just one big tail.  But it does illustrate what happens when there is a lot of probability mass in the bad part of the cost distribution.

I recently ran across this tragic, but beautiful post that illustrates the main point here very well.  The author puts it simply,

    "There are three distinct sides of risk 

  • The odds you will get hit
  • The average consequences of getting hit
  • The tail-end consequences of getting hit" 
The odds of getting hit are $P(r)$, the average consequences are $E(C(r))$ and the tail-end consequences of getting hit are the upper tail of the distribution of $C(r)$.

Monday, August 31, 2020

Five things I look for in Product Managers

SaaS companies live or die by the quality of their product managers.  Great ones really power the future of their companies.  Not-so-great ones waste precious time and investment resources.


It’s very hard to tell from a resume, however, who is going to be a great PM. Great product managers can work for not-so-great companies and vice versa. And what people with this job title actually do varies wildly from company to company.

 

I have been involved in hiring product managers for the past 10+ years. I have hired some great ones and I have made some mistakes. Over the years, I have learned that there five key things that really make a difference:

 

1. Ability to think like someone else

 

Product managers are trained to identify user types or personas. These include information about users’ day-to-day experience, mindset and needs. It’s important to be able to do this, but to be a really great PM, you have to be able to take the next step and actually think like customers, and internal stakeholders as well.

 

I want to see that they can actually put themselves in the position of a user, of a salesperson, a support agent, or other product stakeholder in their company. That they can get into their heads, almost as if they were actors in a play.

 

The best product managers have a tremendous amount of empathy for other people. They can forget about what they think they know about how to use, sell, or support their product and put themselves in the mind of someone who lacks all of that context and has a very different set of day-to-day concerns than they do.

 

During an interview, I test for this by asking the candidate about how users and other stakeholders thought about the products and product development ideas that they worked on. I ask them what surprised them and what they learned from it. It’s not hard to tell who has this ability because those who do exercise it regularly and those who don’t find it hard to come up with examples.

 

2. Determination

 

The most effective product managers are creative and innovative thinkers with positive mindsets. They do not give up. They see the positive outcome as expected and they are surprised by failure. And they see every partial failure as success in need of small refinements - even when there are gaping holes to fill.

 

A lot of today’s most popular products should have died a dozen deaths, but somehow they made it. They made it because their PMs would not give up.

 

Often in an interview, I will give the candidate an example problem of a now-famous payments product that had so many obstacles to overcome in its original conception that people laughed at the idea.

 

I ask the candidate to imagine what it might have been like in the early days of that product, and what problems may have arisen and how they would have tackled them. Their answers show me both their analytical ability and their ability to suspend disbelief and persevere.

 

3. Cultivator of great ideas

 

In most of my product manager interviews, I ask the candidate to tell me some of the top ideas in which they’ve been involved.

 

What I’m looking for here isn’t a detailed explanation of what the person has come up with themselves. Instead, I want to hear about the ideas and products that they helped develop. The best product managers have an uncanny ability to be part of conversations that lead to great ideas. To use a sports analogy, I am less interested in how many points they score individually. Instead, I’m more interested in how much their team scores when they are in the game.

 

To test for this, I ask them to describe the process for developing the top ideas that they mention. I probe to learn about how others were involved and how they helped make sure diverse perspectives and expertise were brought in. If all I hear is “I” and simple logic (talking about done ideas), I know I do not have a great PM in front of me.  Similarly, if all I hear is “we” and I don’t pick up any original contribution from the person I am talking to, I know I am not dealing with a potential game changer.

 

4. Organizing force

 

Great product managers are organizing forces. Note that this is not the same as "being organized." It means that they help define problems in a way that they can be solved with simple solutions and they help develop designs, requirements, and plans so that the teams they are part of can be organized. How product development work is organized and how team members understand the structure of the work is critical. Great PMs can make a night and day difference to the performance of their teams just due to the fact that they help them structure their work better.

 

To test this in an interview, I ask candidates how they would handle a complex product development challenge from my past experience. I ask them to come up with options for how to structure and sequence work, what information they would need to choose among them, and how different stakeholders would understand the approaches.

 

5. Inspiring

 

One of the biggest challenges faced by product managers is how to inspire people who aren’t their direct reports to do things in support of product development efforts. Product managers work across organizations, from engineering to marketing, support, finance, and beyond. It is imperative that they are able to connect and motivate people cross-functionally.

 

In an interview, I will ask them to share past challenges they had with people who had different priorities, and how they were able to motivate the team to get things done. I look for the signs of earned authority - the ability to build followership and commitment to shared goals.


Finding all of these qualities in one person is rare, but I have found that if I probe into each of them I can get a nice well-rounded picture of where a product manager is in their development.  And those who are strong in each of them are a joy to work with and great assets to their teams.

Sunday, March 29, 2020

Put on your own mask first

Difficult times are the ultimate test of leadership. These are the times when lifelong bonds can be established or connections can be lost. Deep human connections are all ultimately based on faith. Faith that through them strength, security and growth will come. Leaders need to ground that faith in confidence, consistency and an unfailing positive attitude.  
At this moment, many leaders have been put in the impossible position of having to show confidence when the foundations of their businesses, their markets and the global economy are being challenged. It doesn’t work to try to hide things or to spin things or to simply put on a happy face. What people need from their leaders today is authentic confidence. That is what leaders have to reach deep down to find right now. The great ones always find it and their connections are strengthened as a result.